[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:39:00 dist-test-slave-dxdd krb5kdc[23959](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:39:00 dist-test-slave-dxdd krb5kdc[23959](info): set up 2 sockets
Sep 25 10:39:00 dist-test-slave-dxdd krb5kdc[23959](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 10:39:01.190587 23952 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Sep 25 10:39:01 dist-test-slave-dxdd krb5kdc[23959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:39:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:39:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
I20260925 10:39:01.363911 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:33209
--webserver_interface=127.23.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:36367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.62:33209
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:39:01.672988 23972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:39:01.673275 23972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:39:01.673367 23972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:39:01.680454 23972 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:39:01.680550 23972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:39:01.680596 23972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:39:01.680653 23972 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:39:01.680691 23972 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:39:01.690151 23972 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:36367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.62:33209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:33209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:39:01.691423 23972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:39:01.692636 23972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:39:01.704134 23978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:01.704190 23977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:01.704713 23972 server_base.cc:1034] running on GCE node
W20260925 10:39:01.704841 23980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:01.705672 23972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:39:01.707104 23972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:39:01.708333 23972 hybrid_clock.cc:648] HybridClock initialized: now 1790332741708286 us; error 58 us; skew 500 ppm
I20260925 10:39:01.708670 23972 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:39:01 dist-test-slave-dxdd krb5kdc[23959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332741, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:39:01.711660 23972 init.cc:377] Logged in from keytab as kudu/127.23.100.62@KRBTEST.COM (short username kudu)
I20260925 10:39:01.713011 23972 webserver.cc:460] Webserver started at http://127.23.100.62:40903/ using document root <none> and password file <none>
I20260925 10:39:01.713687 23972 fs_manager.cc:359] Metadata directory not provided
I20260925 10:39:01.713835 23972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:39:01.714129 23972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:39:01.717006 23972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/master-0/data/instance:
uuid: "2d887d63744547f7b63ecb2793446812"
format_stamp: "Formatted at 2026-09-25 10:39:01 on dist-test-slave-dxdd"
server_key: "5b47a4b99500c8e28a70bf52f2ce3730"
server_key_iv: "5e2e759a69f023b8734637f60f5c7587"
server_key_version: "encryptionkey@0"
I20260925 10:39:01.717774 23972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/master-0/wal/instance:
uuid: "2d887d63744547f7b63ecb2793446812"
format_stamp: "Formatted at 2026-09-25 10:39:01 on dist-test-slave-dxdd"
server_key: "5b47a4b99500c8e28a70bf52f2ce3730"
server_key_iv: "5e2e759a69f023b8734637f60f5c7587"
server_key_version: "encryptionkey@0"
I20260925 10:39:01.722668 23972 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260925 10:39:01.725871 23987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:01.727188 23972 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:39:01.727370 23972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/master-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/master-0/wal
uuid: "2d887d63744547f7b63ecb2793446812"
format_stamp: "Formatted at 2026-09-25 10:39:01 on dist-test-slave-dxdd"
server_key: "5b47a4b99500c8e28a70bf52f2ce3730"
server_key_iv: "5e2e759a69f023b8734637f60f5c7587"
server_key_version: "encryptionkey@0"
I20260925 10:39:01.727571 23972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:39:01.757225 23972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:39:01.761055 23972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:39:01.761343 23972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:39:01.779435 23972 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.62:33209
I20260925 10:39:01.779459 24038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.62:33209 every 8 connection(s)
I20260925 10:39:01.780778 23972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
I20260925 10:39:01.784300 24039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:39:01.787703 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 23972
I20260925 10:39:01.788000 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/master-0/wal/instance
I20260925 10:39:01.789767 23952 external_mini_cluster.cc:1428] Setting key 716d8e93bf2ae2c8a05a9578d8e41d1a
I20260925 10:39:01.795383 24039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d887d63744547f7b63ecb2793446812: Bootstrap starting.
I20260925 10:39:01.798381 24039 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d887d63744547f7b63ecb2793446812: Neither blocks nor log segments found. Creating new log.
I20260925 10:39:01.799391 24039 log.cc:824] T 00000000000000000000000000000000 P 2d887d63744547f7b63ecb2793446812: Log is configured to *not* fsync() on all Append() calls
I20260925 10:39:01.802098 24039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d887d63744547f7b63ecb2793446812: No bootstrap required, opened a new log
I20260925 10:39:01.810407 24039 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d887d63744547f7b63ecb2793446812 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d887d63744547f7b63ecb2793446812" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33209 } }
I20260925 10:39:01.810777 24039 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d887d63744547f7b63ecb2793446812 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:39:01.810881 24039 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d887d63744547f7b63ecb2793446812 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d887d63744547f7b63ecb2793446812, State: Initialized, Role: FOLLOWER
I20260925 10:39:01.811547 24039 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d887d63744547f7b63ecb2793446812 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d887d63744547f7b63ecb2793446812" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33209 } }
I20260925 10:39:01.811817 24039 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d887d63744547f7b63ecb2793446812 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:39:01.811970 24039 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d887d63744547f7b63ecb2793446812 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:39:01.812158 24039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d887d63744547f7b63ecb2793446812 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:39:01.814574 24039 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d887d63744547f7b63ecb2793446812 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d887d63744547f7b63ecb2793446812" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33209 } }
I20260925 10:39:01.815052 24039 leader_election.cc:302] T 00000000000000000000000000000000 P 2d887d63744547f7b63ecb2793446812 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d887d63744547f7b63ecb2793446812; no voters:
I20260925 10:39:01.815641 24039 leader_election.cc:288] T 00000000000000000000000000000000 P 2d887d63744547f7b63ecb2793446812 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:39:01.815797 24044 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d887d63744547f7b63ecb2793446812 [term 1 FOLLOWER]: Leader election won for term 1
Sep 25 10:39:01 dist-test-slave-dxdd krb5kdc[23959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:39:01.816931 24044 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d887d63744547f7b63ecb2793446812 [term 1 LEADER]: Becoming Leader. State: Replica: 2d887d63744547f7b63ecb2793446812, State: Running, Role: LEADER
I20260925 10:39:01.817503 24044 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d887d63744547f7b63ecb2793446812 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d887d63744547f7b63ecb2793446812" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33209 } }
I20260925 10:39:01.817831 24039 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d887d63744547f7b63ecb2793446812 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:39:01.820865 24046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d887d63744547f7b63ecb2793446812 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d887d63744547f7b63ecb2793446812. Latest consensus state: current_term: 1 leader_uuid: "2d887d63744547f7b63ecb2793446812" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d887d63744547f7b63ecb2793446812" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33209 } } }
I20260925 10:39:01.821322 24046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d887d63744547f7b63ecb2793446812 [sys.catalog]: This master's current role is: LEADER
I20260925 10:39:01.822016 24045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d887d63744547f7b63ecb2793446812 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d887d63744547f7b63ecb2793446812" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d887d63744547f7b63ecb2793446812" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33209 } } }
I20260925 10:39:01.822405 24045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d887d63744547f7b63ecb2793446812 [sys.catalog]: This master's current role is: LEADER
I20260925 10:39:01.827904 24051 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:39:01.830530 24041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:01.793507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48528 (local address 127.23.100.62:33209)
0925 10:39:01.794353 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:01.794370 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:01.795496 (+ 1126us) server_negotiation.cc:407] Connection header received
0925 10:39:01.804279 (+ 8783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:01.804298 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:01.804694 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:01.805092 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:01.807977 (+ 2885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:01.809321 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:01.811169 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:01.811937 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:01.812140 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:01.818810 (+ 6670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:01.818824 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:01.818841 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:01.818891 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:01.823634 (+ 4743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:01.824912 (+ 1278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:01.824924 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:01.824935 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:01.825194 (+ 259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:01.825872 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:01.825877 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:01.825880 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:01.826659 (+ 779us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:01.826979 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:01.828213 (+ 1234us) server_negotiation.cc:299] Negotiation successful
0925 10:39:01.828740 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":394,"threads_started":1}
I20260925 10:39:01.833691 24051 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:39:01.842850 24051 catalog_manager.cc:1347] Generated new cluster ID: 24e2d12b371049a5a83691eb2d1b4ecd
I20260925 10:39:01.842978 24051 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:39:01.872586 24051 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:39:01.874449 24051 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:39:01.884318 24051 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d887d63744547f7b63ecb2793446812: Generated new TSK 0
I20260925 10:39:01.885093 24051 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab.
Entry for principal kudu/127.23.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab.
I20260925 10:39:01.939623 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.1:0
--local_ip_for_outbound_sockets=127.23.100.1
--webserver_interface=127.23.100.1
--webserver_port=0
--tserver_master_addrs=127.23.100.62:33209
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab
--principal=kudu/127.23.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:36367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:39:02.275271 24065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:39:02.275584 24065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:39:02.275758 24065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:39:02.282778 24065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:39:02.282922 24065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:39:02.283108 24065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.1
I20260925 10:39:02.292397 24065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:36367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab
--principal=kudu/127.23.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.1
--webserver_port=0
--tserver_master_addrs=127.23.100.62:33209
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.1
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:39:02.293627 24065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:39:02.294955 24065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:39:02.308022 24070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:02.307937 24071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:02.307992 24073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:02.308743 24065 server_base.cc:1034] running on GCE node
I20260925 10:39:02.309306 24065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:39:02.310165 24065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:39:02.311391 24065 hybrid_clock.cc:648] HybridClock initialized: now 1790332742311340 us; error 67 us; skew 500 ppm
I20260925 10:39:02.311688 24065 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:39:02 dist-test-slave-dxdd krb5kdc[23959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332742, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:39:02.314742 24065 init.cc:377] Logged in from keytab as kudu/127.23.100.1@KRBTEST.COM (short username kudu)
I20260925 10:39:02.316135 24065 webserver.cc:460] Webserver started at http://127.23.100.1:37945/ using document root <none> and password file <none>
I20260925 10:39:02.316790 24065 fs_manager.cc:359] Metadata directory not provided
I20260925 10:39:02.316931 24065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:39:02.317248 24065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:39:02.320358 24065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/ts-0/data/instance:
uuid: "031a0a37a2964eee8c9a72e90b5d6dac"
format_stamp: "Formatted at 2026-09-25 10:39:02 on dist-test-slave-dxdd"
server_key: "c3d112bf3a3296ba573eb1fac462c05d"
server_key_iv: "02ee3df6acb41791b5adfc4c0832dc20"
server_key_version: "encryptionkey@0"
I20260925 10:39:02.321094 24065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/ts-0/wal/instance:
uuid: "031a0a37a2964eee8c9a72e90b5d6dac"
format_stamp: "Formatted at 2026-09-25 10:39:02 on dist-test-slave-dxdd"
server_key: "c3d112bf3a3296ba573eb1fac462c05d"
server_key_iv: "02ee3df6acb41791b5adfc4c0832dc20"
server_key_version: "encryptionkey@0"
I20260925 10:39:02.326136 24065 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 10:39:02.329877 24080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:02.331463 24065 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:39:02.331633 24065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/ts-0/wal
uuid: "031a0a37a2964eee8c9a72e90b5d6dac"
format_stamp: "Formatted at 2026-09-25 10:39:02 on dist-test-slave-dxdd"
server_key: "c3d112bf3a3296ba573eb1fac462c05d"
server_key_iv: "02ee3df6acb41791b5adfc4c0832dc20"
server_key_version: "encryptionkey@0"
I20260925 10:39:02.331867 24065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:39:02.346019 24065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:39:02.349996 24065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:39:02.350296 24065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:39:02.351236 24065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:39:02.352605 24065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:39:02.352700 24065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:02.352828 24065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:39:02.352880 24065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:02.385823 24065 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.1:46793
I20260925 10:39:02.385869 24192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.1:46793 every 8 connection(s)
I20260925 10:39:02.387192 24065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
I20260925 10:39:02.387645 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 24065
I20260925 10:39:02.387868 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/ts-0/wal/instance
I20260925 10:39:02.388680 23952 external_mini_cluster.cc:1428] Setting key e9fb38951018bc907d149bd0ee48ea77
Sep 25 10:39:02 dist-test-slave-dxdd krb5kdc[23959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332742, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.1@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.100.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.2@KRBTEST.COM" created.
I20260925 10:39:02.411319 24195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:02.389896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:40445 (local address 127.23.100.62:33209)
0925 10:39:02.390410 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:02.390416 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:02.391647 (+ 1231us) server_negotiation.cc:407] Connection header received
0925 10:39:02.394787 (+ 3140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:02.394792 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:02.394856 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:02.394940 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:02.396897 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:02.397847 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:02.400537 (+ 2690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:02.400926 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:02.401026 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:02.405279 (+ 4253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:02.405284 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:02.405287 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:02.405319 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:02.407983 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:02.408765 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:02.408769 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:02.408772 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:02.408923 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:02.409372 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:02.409375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:02.409377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:02.409734 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:02.409896 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:02.410904 (+ 1008us) server_negotiation.cc:299] Negotiation successful
0925 10:39:02.411074 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":284,"threads_started":1}
I20260925 10:39:02.413012 24196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:02.390598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:33209 (local address 127.23.100.1:40445)
0925 10:39:02.391431 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:02.391494 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:02.394472 (+ 2978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:02.395238 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:02.395251 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:02.396049 (+ 798us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:02.396720 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:02.396743 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:02.398011 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:02.398020 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:02.400398 (+ 2378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:02.400417 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:02.401167 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:02.401176 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:02.401499 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:02.402265 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:02.402300 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:02.405059 (+ 2759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:02.408163 (+ 3104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:02.408174 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:02.408194 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:02.408632 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:02.409054 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:02.409057 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:02.409060 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:02.409261 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:02.409875 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:02.409882 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:02.410364 (+ 482us) client_negotiation.cc:769] Sending connection context
0925 10:39:02.410791 (+ 427us) client_negotiation.cc:240] Negotiation successful
0925 10:39:02.411384 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":292,"threads_started":1}
I20260925 10:39:02.414585 24193 heartbeater.cc:347] Connected to a master server at 127.23.100.62:33209
I20260925 10:39:02.414953 24193 heartbeater.cc:464] Registering TS with master...
I20260925 10:39:02.415874 24193 heartbeater.cc:511] Master 127.23.100.62:33209 requested a full tablet report, sending...
I20260925 10:39:02.418110 24004 ts_manager.cc:194] Registered new tserver with Master: 031a0a37a2964eee8c9a72e90b5d6dac (127.23.100.1:46793)
I20260925 10:39:02.419682 24004 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.1@KRBTEST.COM'} at 127.23.100.1:40445
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab.
Entry for principal kudu/127.23.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab.
I20260925 10:39:02.433950 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.2:0
--local_ip_for_outbound_sockets=127.23.100.2
--webserver_interface=127.23.100.2
--webserver_port=0
--tserver_master_addrs=127.23.100.62:33209
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab
--principal=kudu/127.23.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:36367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:39:02.748723 24199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:39:02.749046 24199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:39:02.749225 24199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:39:02.756548 24199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:39:02.756702 24199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:39:02.756889 24199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.2
I20260925 10:39:02.766572 24199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:36367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab
--principal=kudu/127.23.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.2
--webserver_port=0
--tserver_master_addrs=127.23.100.62:33209
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.2
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:39:02.767792 24199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:39:02.769114 24199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:39:02.781716 24204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:02.781764 24205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:02.782459 24207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:02.782902 24199 server_base.cc:1034] running on GCE node
I20260925 10:39:02.783541 24199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:39:02.784543 24199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:39:02.785768 24199 hybrid_clock.cc:648] HybridClock initialized: now 1790332742785714 us; error 45 us; skew 500 ppm
I20260925 10:39:02.786087 24199 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:39:02 dist-test-slave-dxdd krb5kdc[23959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332742, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:39:02.789747 24199 init.cc:377] Logged in from keytab as kudu/127.23.100.2@KRBTEST.COM (short username kudu)
I20260925 10:39:02.797284 24199 webserver.cc:460] Webserver started at http://127.23.100.2:45245/ using document root <none> and password file <none>
I20260925 10:39:02.797928 24199 fs_manager.cc:359] Metadata directory not provided
I20260925 10:39:02.798074 24199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:39:02.798473 24199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:39:02.801205 24199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/ts-1/data/instance:
uuid: "e8d28cdd2d1c4a35a242a04f7c643114"
format_stamp: "Formatted at 2026-09-25 10:39:02 on dist-test-slave-dxdd"
server_key: "a135b77cf94845ff531d7569d91ef855"
server_key_iv: "185df020169544a19d563b3c4e1553a5"
server_key_version: "encryptionkey@0"
I20260925 10:39:02.801939 24199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/ts-1/wal/instance:
uuid: "e8d28cdd2d1c4a35a242a04f7c643114"
format_stamp: "Formatted at 2026-09-25 10:39:02 on dist-test-slave-dxdd"
server_key: "a135b77cf94845ff531d7569d91ef855"
server_key_iv: "185df020169544a19d563b3c4e1553a5"
server_key_version: "encryptionkey@0"
I20260925 10:39:02.806828 24199 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260925 10:39:02.809964 24214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:02.811167 24199 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 10:39:02.811344 24199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/ts-1/wal
uuid: "e8d28cdd2d1c4a35a242a04f7c643114"
format_stamp: "Formatted at 2026-09-25 10:39:02 on dist-test-slave-dxdd"
server_key: "a135b77cf94845ff531d7569d91ef855"
server_key_iv: "185df020169544a19d563b3c4e1553a5"
server_key_version: "encryptionkey@0"
I20260925 10:39:02.811532 24199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:39:02.841534 24199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:39:02.845556 24199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:39:02.845840 24199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:39:02.846861 24199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:39:02.848232 24199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:39:02.848336 24199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:02.848480 24199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:39:02.848559 24199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:02.882015 24199 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.2:38651
I20260925 10:39:02.882061 24326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.2:38651 every 8 connection(s)
I20260925 10:39:02.883358 24199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
I20260925 10:39:02.890002 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 24199
I20260925 10:39:02.890221 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/ts-1/wal/instance
I20260925 10:39:02.890697 23952 external_mini_cluster.cc:1428] Setting key 8b1f9d56d3626fd579375f43f334d27f
Sep 25 10:39:02 dist-test-slave-dxdd krb5kdc[23959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332742, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.2@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:39:02.903328 24195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:02.885954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:33733 (local address 127.23.100.62:33209)
0925 10:39:02.886137 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:02.886142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:02.887391 (+ 1249us) server_negotiation.cc:407] Connection header received
0925 10:39:02.889167 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:02.889172 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:02.889263 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:02.889373 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:02.890970 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:02.891808 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:02.893473 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:02.893887 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:02.894019 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:02.897539 (+ 3520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:02.897552 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:02.897556 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:02.897585 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:02.900142 (+ 2557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:02.901043 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:02.901047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:02.901049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:02.901164 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:02.901635 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:02.901638 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:02.901640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:02.901876 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:02.901981 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:02.902895 (+ 914us) server_negotiation.cc:299] Negotiation successful
0925 10:39:02.903104 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 10:39:02.904819 24329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:02.886554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:33209 (local address 127.23.100.2:33733)
0925 10:39:02.887232 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:02.887273 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:02.888913 (+ 1640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:02.889600 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:02.889608 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:02.890155 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:02.890799 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:02.890823 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:02.891953 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:02.891958 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:02.893333 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:02.893344 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:02.894105 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:02.894112 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:02.894396 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:02.895055 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:02.895076 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:02.897322 (+ 2246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:02.900285 (+ 2963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:02.900296 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:02.900314 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:02.900922 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:02.901279 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:02.901287 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:02.901293 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:02.901526 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:02.902001 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:02.902012 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:02.902474 (+ 462us) client_negotiation.cc:769] Sending connection context
0925 10:39:02.902780 (+ 306us) client_negotiation.cc:240] Negotiation successful
0925 10:39:02.903334 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":270,"threads_started":1}
I20260925 10:39:02.906200 24327 heartbeater.cc:347] Connected to a master server at 127.23.100.62:33209
I20260925 10:39:02.906543 24327 heartbeater.cc:464] Registering TS with master...
I20260925 10:39:02.907230 24327 heartbeater.cc:511] Master 127.23.100.62:33209 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.100.3@KRBTEST.COM; defaulting to no policy
I20260925 10:39:02.908560 24004 ts_manager.cc:194] Registered new tserver with Master: e8d28cdd2d1c4a35a242a04f7c643114 (127.23.100.2:38651)
I20260925 10:39:02.909404 24004 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.2@KRBTEST.COM'} at 127.23.100.2:33733
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab.
Entry for principal kudu/127.23.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab.
I20260925 10:39:02.931557 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.3:0
--local_ip_for_outbound_sockets=127.23.100.3
--webserver_interface=127.23.100.3
--webserver_port=0
--tserver_master_addrs=127.23.100.62:33209
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab
--principal=kudu/127.23.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:36367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:39:03.250384 24332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:39:03.250646 24332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:39:03.250802 24332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:39:03.257898 24332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:39:03.258020 24332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:39:03.258206 24332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.3
I20260925 10:39:03.267560 24332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:36367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab
--principal=kudu/127.23.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.3
--webserver_port=0
--tserver_master_addrs=127.23.100.62:33209
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.3
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:39:03.268760 24332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:39:03.270015 24332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:39:03.282640 24337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:03.282594 24338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:03.282676 24340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:03.283347 24332 server_base.cc:1034] running on GCE node
I20260925 10:39:03.283859 24332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:39:03.284649 24332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:39:03.285871 24332 hybrid_clock.cc:648] HybridClock initialized: now 1790332743285812 us; error 48 us; skew 500 ppm
I20260925 10:39:03.286154 24332 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:39:03 dist-test-slave-dxdd krb5kdc[23959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332743, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:39:03.289045 24332 init.cc:377] Logged in from keytab as kudu/127.23.100.3@KRBTEST.COM (short username kudu)
I20260925 10:39:03.290266 24332 webserver.cc:460] Webserver started at http://127.23.100.3:36781/ using document root <none> and password file <none>
I20260925 10:39:03.290901 24332 fs_manager.cc:359] Metadata directory not provided
I20260925 10:39:03.290997 24332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:39:03.291232 24332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:39:03.293879 24332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/ts-2/data/instance:
uuid: "d6eb6317b23647c2964c3fa8764cdb0b"
format_stamp: "Formatted at 2026-09-25 10:39:03 on dist-test-slave-dxdd"
server_key: "0bcff1fe4a36c25e361722b44ed92c39"
server_key_iv: "1ba9c2002c0ba321865c4ed90b829881"
server_key_version: "encryptionkey@0"
I20260925 10:39:03.294612 24332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/ts-2/wal/instance:
uuid: "d6eb6317b23647c2964c3fa8764cdb0b"
format_stamp: "Formatted at 2026-09-25 10:39:03 on dist-test-slave-dxdd"
server_key: "0bcff1fe4a36c25e361722b44ed92c39"
server_key_iv: "1ba9c2002c0ba321865c4ed90b829881"
server_key_version: "encryptionkey@0"
I20260925 10:39:03.299321 24332 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 10:39:03.302556 24347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:03.303836 24332 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 10:39:03.303956 24332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/ts-2/wal
uuid: "d6eb6317b23647c2964c3fa8764cdb0b"
format_stamp: "Formatted at 2026-09-25 10:39:03 on dist-test-slave-dxdd"
server_key: "0bcff1fe4a36c25e361722b44ed92c39"
server_key_iv: "1ba9c2002c0ba321865c4ed90b829881"
server_key_version: "encryptionkey@0"
I20260925 10:39:03.304096 24332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:39:03.322551 24332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:39:03.326246 24332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:39:03.326565 24332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:39:03.327601 24332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:39:03.328963 24332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:39:03.329033 24332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:03.329190 24332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:39:03.329239 24332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:03.361523 24332 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.3:39733
I20260925 10:39:03.361568 24459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.3:39733 every 8 connection(s)
I20260925 10:39:03.362818 24332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
I20260925 10:39:03.367237 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 24332
I20260925 10:39:03.367405 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790332740062128-23952-0/minicluster-data/ts-2/wal/instance
I20260925 10:39:03.367815 23952 external_mini_cluster.cc:1428] Setting key 21e5dbd4601ce8741c3d089e64f30613
Sep 25 10:39:03 dist-test-slave-dxdd krb5kdc[23959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332743, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.3@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:39:03.381515 24195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:03.365408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:39143 (local address 127.23.100.62:33209)
0925 10:39:03.365582 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:03.365586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:03.366779 (+ 1193us) server_negotiation.cc:407] Connection header received
0925 10:39:03.368615 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:03.368620 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:03.368712 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:03.368809 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:03.370483 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:03.371491 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:03.372861 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:03.373246 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:03.373334 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:03.376396 (+ 3062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:03.376401 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:03.376404 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:03.376433 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:03.379037 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:03.379659 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:03.379662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:03.379664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:03.379790 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:03.380143 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:03.380146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:03.380148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:03.380377 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:03.380501 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:03.381117 (+ 616us) server_negotiation.cc:299] Negotiation successful
0925 10:39:03.381269 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 10:39:03.382789 24462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:03.365891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:33209 (local address 127.23.100.3:39143)
0925 10:39:03.366626 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:03.366673 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:03.368374 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:03.369059 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:03.369066 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:03.369655 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:03.370292 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:03.370357 (+ 65us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:03.371610 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:03.371614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:03.372754 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:03.372765 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:03.373414 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:03.373419 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:03.373601 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:03.374177 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:03.374200 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:03.376239 (+ 2039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:03.379151 (+ 2912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:03.379157 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:03.379168 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:03.379560 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:03.379883 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:03.379886 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:03.379888 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:03.380050 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:03.380509 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:03.380515 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:03.380811 (+ 296us) client_negotiation.cc:769] Sending connection context
0925 10:39:03.381024 (+ 213us) client_negotiation.cc:240] Negotiation successful
0925 10:39:03.381372 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":267,"threads_started":1}
I20260925 10:39:03.384151 24460 heartbeater.cc:347] Connected to a master server at 127.23.100.62:33209
I20260925 10:39:03.384430 24460 heartbeater.cc:464] Registering TS with master...
I20260925 10:39:03.385131 24460 heartbeater.cc:511] Master 127.23.100.62:33209 requested a full tablet report, sending...
I20260925 10:39:03.386579 24004 ts_manager.cc:194] Registered new tserver with Master: d6eb6317b23647c2964c3fa8764cdb0b (127.23.100.3:39733)
I20260925 10:39:03.387337 24004 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.3@KRBTEST.COM'} at 127.23.100.3:39143
I20260925 10:39:03.395674 23952 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 10:39:03 dist-test-slave-dxdd krb5kdc[23959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332743, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260925 10:39:03.423477 24193 heartbeater.cc:503] Master 127.23.100.62:33209 was elected leader, sending a full tablet report...
Sep 25 10:39:03 dist-test-slave-dxdd krb5kdc[23959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332743, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.100.1@KRBTEST.COM
I20260925 10:39:03.438412 24467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:03.423961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43956 (local address 127.23.100.1:46793)
0925 10:39:03.424650 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:03.424659 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:03.424698 (+ 39us) server_negotiation.cc:407] Connection header received
0925 10:39:03.424770 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:03.424776 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:03.425062 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:03.425237 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:03.426687 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:03.427759 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:03.429084 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:03.429615 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:03.429742 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:03.433509 (+ 3767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:03.433518 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:03.433531 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:03.433563 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:03.436127 (+ 2564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:03.436707 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:03.436714 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:03.436716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:03.436826 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:03.437151 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:03.437154 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:03.437156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:03.437465 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:03.437679 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:03.437896 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 10:39:03.438073 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":810,"threads_started":1}
W20260925 10:39:03.439546 24108 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43956
Sep 25 10:39:03 dist-test-slave-dxdd krb5kdc[23959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 10:39:03 dist-test-slave-dxdd krb5kdc[23959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.1@KRBTEST.COM
I20260925 10:39:03.479172 24467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:03.466937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43970 (local address 127.23.100.1:46793)
0925 10:39:03.467140 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:03.467146 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:03.467325 (+ 179us) server_negotiation.cc:407] Connection header received
0925 10:39:03.467480 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:03.467484 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:03.467543 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:03.467618 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:03.468816 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:03.469453 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:03.470626 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:03.471043 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:03.471135 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:03.474610 (+ 3475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:03.474615 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:03.474618 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:03.474647 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:03.477125 (+ 2478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:03.477651 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:03.477654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:03.477657 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:03.477774 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:03.478054 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:03.478057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:03.478059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:03.478270 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:03.478490 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:03.478709 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 10:39:03.478857 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 10:39:03.480481 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 24065
I20260925 10:39:03.491997 24467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:03.481309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43986 (local address 127.23.100.1:46793)
0925 10:39:03.481488 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:03.481497 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:03.481776 (+ 279us) server_negotiation.cc:407] Connection header received
0925 10:39:03.481867 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:03.481870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:03.481916 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:03.481986 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:03.483105 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:03.483761 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:03.484906 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:03.485289 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:03.485372 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:03.487380 (+ 2008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:03.487384 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:03.487387 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:03.487414 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:03.489870 (+ 2456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:03.490498 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:03.490502 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:03.490504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:03.490623 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:03.490921 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:03.490923 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:03.490925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:03.491164 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:03.491327 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:03.491515 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 10:39:03.491659 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 10:39:03.492643 24188 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:39:03.574978 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 24065
I20260925 10:39:03.591178 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 24199
Sep 25 10:39:03 dist-test-slave-dxdd krb5kdc[23959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.2@KRBTEST.COM
I20260925 10:39:03.606492 24474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:03.592025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60032 (local address 127.23.100.2:38651)
0925 10:39:03.592570 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:03.592577 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:03.592602 (+ 25us) server_negotiation.cc:407] Connection header received
0925 10:39:03.592661 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:03.592668 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:03.592886 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:03.593014 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:03.594109 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:03.595136 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:03.596310 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:03.596816 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:03.596910 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:03.600954 (+ 4044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:03.600965 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:03.600977 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:03.601007 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:03.603743 (+ 2736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:03.604340 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:03.604346 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:03.604348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:03.604463 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:03.604839 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:03.604844 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:03.604851 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:03.605170 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:03.605465 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:03.605789 (+ 324us) server_negotiation.cc:299] Negotiation successful
0925 10:39:03.606063 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":272,"threads_started":1}
I20260925 10:39:03.607755 24322 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:39:03.690510 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 24199
I20260925 10:39:03.705906 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 24332
Sep 25 10:39:03 dist-test-slave-dxdd krb5kdc[23959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.3@KRBTEST.COM
I20260925 10:39:03.721995 24476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:03.706916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53318 (local address 127.23.100.3:39733)
0925 10:39:03.707525 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:03.707531 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:03.707558 (+ 27us) server_negotiation.cc:407] Connection header received
0925 10:39:03.707747 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:03.707755 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:03.707984 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:03.708162 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:03.709349 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:03.710466 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:03.711725 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:03.712279 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:03.712415 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:03.716512 (+ 4097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:03.716523 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:03.716536 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:03.716569 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:03.719329 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:03.719986 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:03.719994 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:03.719996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:03.720114 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:03.720462 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:03.720469 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:03.720472 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:03.720794 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:03.721100 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:03.721393 (+ 293us) server_negotiation.cc:299] Negotiation successful
0925 10:39:03.721661 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":307,"threads_started":1}
I20260925 10:39:03.723287 24455 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:39:03.808180 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 24332
I20260925 10:39:03.823607 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 23972
I20260925 10:39:03.824747 24034 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:39:03.926991 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 23972
2026-09-25T10:39:03Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3831 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:39:04 dist-test-slave-dxdd krb5kdc[24481](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:39:04 dist-test-slave-dxdd krb5kdc[24481](info): set up 2 sockets
Sep 25 10:39:04 dist-test-slave-dxdd krb5kdc[24481](info): commencing operation
krb5kdc: starting...
W20260925 10:39:04.998239 23952 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Sep 25 10:39:05 dist-test-slave-dxdd krb5kdc[24481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:39:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:39:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
I20260925 10:39:05.179838 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:35229
--webserver_interface=127.23.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:38145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.62:35229
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:39:05.484968 24494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:39:05.485268 24494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:39:05.485435 24494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:39:05.492388 24494 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:39:05.492511 24494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:39:05.492607 24494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:39:05.492695 24494 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:39:05.492781 24494 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:39:05.502172 24494 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:38145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.62:35229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:35229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:39:05.503391 24494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:39:05.504647 24494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:39:05.516049 24502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:05.516098 24499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:05.516202 24500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:05.517316 24494 server_base.cc:1034] running on GCE node
I20260925 10:39:05.517997 24494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:39:05.519280 24494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:39:05.520467 24494 hybrid_clock.cc:648] HybridClock initialized: now 1790332745520430 us; error 34 us; skew 500 ppm
I20260925 10:39:05.520792 24494 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:39:05 dist-test-slave-dxdd krb5kdc[24481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332745, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:39:05.528975 24494 init.cc:377] Logged in from keytab as kudu/127.23.100.62@KRBTEST.COM (short username kudu)
I20260925 10:39:05.530395 24494 webserver.cc:460] Webserver started at http://127.23.100.62:46565/ using document root <none> and password file <none>
I20260925 10:39:05.531037 24494 fs_manager.cc:359] Metadata directory not provided
I20260925 10:39:05.531162 24494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:39:05.531467 24494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:39:05.534343 24494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/master-0/data/instance:
uuid: "bc492321ecc54cf99b031d831d6547b8"
format_stamp: "Formatted at 2026-09-25 10:39:05 on dist-test-slave-dxdd"
server_key: "f24bda930c005bb25d90403e0d375695"
server_key_iv: "14a6879b1f879865f55242e736643cef"
server_key_version: "encryptionkey@0"
I20260925 10:39:05.535092 24494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/master-0/wal/instance:
uuid: "bc492321ecc54cf99b031d831d6547b8"
format_stamp: "Formatted at 2026-09-25 10:39:05 on dist-test-slave-dxdd"
server_key: "f24bda930c005bb25d90403e0d375695"
server_key_iv: "14a6879b1f879865f55242e736643cef"
server_key_version: "encryptionkey@0"
I20260925 10:39:05.539856 24494 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 10:39:05.542950 24509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:05.544078 24494 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 10:39:05.544242 24494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/master-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/master-0/wal
uuid: "bc492321ecc54cf99b031d831d6547b8"
format_stamp: "Formatted at 2026-09-25 10:39:05 on dist-test-slave-dxdd"
server_key: "f24bda930c005bb25d90403e0d375695"
server_key_iv: "14a6879b1f879865f55242e736643cef"
server_key_version: "encryptionkey@0"
I20260925 10:39:05.544412 24494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:39:05.557916 24494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:39:05.561622 24494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:39:05.561887 24494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:39:05.579131 24494 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.62:35229
I20260925 10:39:05.579150 24560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.62:35229 every 8 connection(s)
I20260925 10:39:05.580420 24494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
I20260925 10:39:05.584457 24561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:39:05.584657 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 24494
I20260925 10:39:05.584888 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/master-0/wal/instance
I20260925 10:39:05.585721 23952 external_mini_cluster.cc:1428] Setting key d861f0b9262a719877ba6a14271d7cbf
I20260925 10:39:05.598109 24561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc492321ecc54cf99b031d831d6547b8: Bootstrap starting.
Sep 25 10:39:05 dist-test-slave-dxdd krb5kdc[24481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:39:05.601549 24561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc492321ecc54cf99b031d831d6547b8: Neither blocks nor log segments found. Creating new log.
I20260925 10:39:05.602800 24561 log.cc:824] T 00000000000000000000000000000000 P bc492321ecc54cf99b031d831d6547b8: Log is configured to *not* fsync() on all Append() calls
I20260925 10:39:05.605365 24561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc492321ecc54cf99b031d831d6547b8: No bootstrap required, opened a new log
I20260925 10:39:05.608048 24564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:05.587738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43776 (local address 127.23.100.62:35229)
0925 10:39:05.588469 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:05.588479 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:05.588520 (+ 41us) server_negotiation.cc:407] Connection header received
0925 10:39:05.590572 (+ 2052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:05.590593 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:05.590957 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:05.591267 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:05.593002 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:05.594242 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:05.596028 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:05.596646 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:05.596767 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:05.601193 (+ 4426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:05.601201 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:05.601214 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:05.601245 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:05.604254 (+ 3009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:05.604871 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:05.604878 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:05.604883 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:05.605036 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:05.605380 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:05.605386 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:05.605389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:05.605913 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:05.606210 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:05.606721 (+ 511us) server_negotiation.cc:299] Negotiation successful
0925 10:39:05.607123 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":347,"threads_started":1}
I20260925 10:39:05.613286 24561 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc492321ecc54cf99b031d831d6547b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc492321ecc54cf99b031d831d6547b8" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 35229 } }
I20260925 10:39:05.613580 24561 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc492321ecc54cf99b031d831d6547b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:39:05.613662 24561 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc492321ecc54cf99b031d831d6547b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc492321ecc54cf99b031d831d6547b8, State: Initialized, Role: FOLLOWER
I20260925 10:39:05.614137 24561 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc492321ecc54cf99b031d831d6547b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc492321ecc54cf99b031d831d6547b8" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 35229 } }
I20260925 10:39:05.614375 24561 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc492321ecc54cf99b031d831d6547b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:39:05.614524 24561 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc492321ecc54cf99b031d831d6547b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:39:05.614674 24561 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc492321ecc54cf99b031d831d6547b8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:39:05.616982 24561 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc492321ecc54cf99b031d831d6547b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc492321ecc54cf99b031d831d6547b8" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 35229 } }
I20260925 10:39:05.617478 24561 leader_election.cc:302] T 00000000000000000000000000000000 P bc492321ecc54cf99b031d831d6547b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc492321ecc54cf99b031d831d6547b8; no voters:
I20260925 10:39:05.617947 24561 leader_election.cc:288] T 00000000000000000000000000000000 P bc492321ecc54cf99b031d831d6547b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:39:05.618072 24566 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc492321ecc54cf99b031d831d6547b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:39:05.618731 24566 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc492321ecc54cf99b031d831d6547b8 [term 1 LEADER]: Becoming Leader. State: Replica: bc492321ecc54cf99b031d831d6547b8, State: Running, Role: LEADER
I20260925 10:39:05.619192 24566 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc492321ecc54cf99b031d831d6547b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc492321ecc54cf99b031d831d6547b8" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 35229 } }
I20260925 10:39:05.619840 24561 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc492321ecc54cf99b031d831d6547b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:39:05.621706 24567 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc492321ecc54cf99b031d831d6547b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc492321ecc54cf99b031d831d6547b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc492321ecc54cf99b031d831d6547b8" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 35229 } } }
I20260925 10:39:05.622094 24567 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc492321ecc54cf99b031d831d6547b8 [sys.catalog]: This master's current role is: LEADER
I20260925 10:39:05.622500 24568 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc492321ecc54cf99b031d831d6547b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc492321ecc54cf99b031d831d6547b8. Latest consensus state: current_term: 1 leader_uuid: "bc492321ecc54cf99b031d831d6547b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc492321ecc54cf99b031d831d6547b8" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 35229 } } }
I20260925 10:39:05.622759 24568 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc492321ecc54cf99b031d831d6547b8 [sys.catalog]: This master's current role is: LEADER
I20260925 10:39:05.624432 24573 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:39:05.629118 24573 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:39:05.637871 24573 catalog_manager.cc:1347] Generated new cluster ID: fca99e5786bf4b4f8478dba57b509199
I20260925 10:39:05.637967 24573 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:39:05.680635 24573 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:39:05.682535 24573 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:39:05.694864 24573 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc492321ecc54cf99b031d831d6547b8: Generated new TSK 0
I20260925 10:39:05.695796 24573 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab.
Entry for principal kudu/127.23.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab.
I20260925 10:39:05.747677 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.1:0
--local_ip_for_outbound_sockets=127.23.100.1
--webserver_interface=127.23.100.1
--webserver_port=0
--tserver_master_addrs=127.23.100.62:35229
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab
--principal=kudu/127.23.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:38145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:39:06.064265 24587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:39:06.064528 24587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:39:06.064705 24587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:39:06.072045 24587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:39:06.072173 24587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:39:06.072362 24587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.1
I20260925 10:39:06.081952 24587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:38145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab
--principal=kudu/127.23.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.1
--webserver_port=0
--tserver_master_addrs=127.23.100.62:35229
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.1
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:39:06.083246 24587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:39:06.084501 24587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:39:06.096864 24592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:06.096935 24595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:06.096963 24593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:06.097617 24587 server_base.cc:1034] running on GCE node
I20260925 10:39:06.098168 24587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:39:06.099094 24587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:39:06.100324 24587 hybrid_clock.cc:648] HybridClock initialized: now 1790332746100284 us; error 40 us; skew 500 ppm
I20260925 10:39:06.100649 24587 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:39:06 dist-test-slave-dxdd krb5kdc[24481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332746, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:39:06.103645 24587 init.cc:377] Logged in from keytab as kudu/127.23.100.1@KRBTEST.COM (short username kudu)
I20260925 10:39:06.105037 24587 webserver.cc:460] Webserver started at http://127.23.100.1:35787/ using document root <none> and password file <none>
I20260925 10:39:06.105686 24587 fs_manager.cc:359] Metadata directory not provided
I20260925 10:39:06.105822 24587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:39:06.106112 24587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:39:06.109371 24587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/ts-0/data/instance:
uuid: "ffc4c9bfd37a4e2b8db9899e938e87bd"
format_stamp: "Formatted at 2026-09-25 10:39:06 on dist-test-slave-dxdd"
server_key: "72150e3767005591ed37b34a911498a3"
server_key_iv: "57dcb403c64fadbbd23ae74fde1cabc3"
server_key_version: "encryptionkey@0"
I20260925 10:39:06.110126 24587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/ts-0/wal/instance:
uuid: "ffc4c9bfd37a4e2b8db9899e938e87bd"
format_stamp: "Formatted at 2026-09-25 10:39:06 on dist-test-slave-dxdd"
server_key: "72150e3767005591ed37b34a911498a3"
server_key_iv: "57dcb403c64fadbbd23ae74fde1cabc3"
server_key_version: "encryptionkey@0"
I20260925 10:39:06.115234 24587 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 10:39:06.118579 24602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:06.119801 24587 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 10:39:06.119988 24587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/ts-0/wal
uuid: "ffc4c9bfd37a4e2b8db9899e938e87bd"
format_stamp: "Formatted at 2026-09-25 10:39:06 on dist-test-slave-dxdd"
server_key: "72150e3767005591ed37b34a911498a3"
server_key_iv: "57dcb403c64fadbbd23ae74fde1cabc3"
server_key_version: "encryptionkey@0"
I20260925 10:39:06.120201 24587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:39:06.133996 24587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:39:06.138253 24587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:39:06.138556 24587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:39:06.139482 24587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:39:06.140726 24587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:39:06.140830 24587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:06.140962 24587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:39:06.141034 24587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:06.173106 24587 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.1:43993
I20260925 10:39:06.173153 24714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.1:43993 every 8 connection(s)
I20260925 10:39:06.174772 24587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
I20260925 10:39:06.183992 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 24587
I20260925 10:39:06.184206 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/ts-0/wal/instance
I20260925 10:39:06.184870 23952 external_mini_cluster.cc:1428] Setting key 583f241d4d2a7fbbc71d9960bb3eb289
Sep 25 10:39:06 dist-test-slave-dxdd krb5kdc[24481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332746, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.1@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:39:06.193764 24717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:06.177214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:37773 (local address 127.23.100.62:35229)
0925 10:39:06.177652 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:06.177656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:06.178527 (+ 871us) server_negotiation.cc:407] Connection header received
0925 10:39:06.180320 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:06.180326 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:06.180423 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:06.180534 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:06.182011 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:06.182796 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:06.184439 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:06.184834 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:06.184955 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:06.188394 (+ 3439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:06.188399 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:06.188402 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:06.188432 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:06.190996 (+ 2564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:06.191657 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:06.191664 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:06.191667 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:06.191799 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:06.192191 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:06.192197 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:06.192201 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:06.192453 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:06.192603 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:06.193217 (+ 614us) server_negotiation.cc:299] Negotiation successful
0925 10:39:06.193423 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":239,"threads_started":1}
I20260925 10:39:06.194908 24718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:06.177639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35229 (local address 127.23.100.1:37773)
0925 10:39:06.178378 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:06.178422 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:06.180042 (+ 1620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:06.180781 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:06.180791 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:06.181298 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:06.181846 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:06.181864 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:06.182943 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:06.182947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:06.184293 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:06.184310 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:06.185049 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:06.185055 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:06.185273 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:06.185981 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:06.186009 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:06.188175 (+ 2166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:06.191127 (+ 2952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:06.191135 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:06.191151 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:06.191544 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:06.191903 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:06.191908 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:06.191911 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:06.192086 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:06.192574 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:06.192581 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:06.192898 (+ 317us) client_negotiation.cc:769] Sending connection context
0925 10:39:06.193143 (+ 245us) client_negotiation.cc:240] Negotiation successful
0925 10:39:06.193641 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":270,"threads_started":1}
I20260925 10:39:06.196393 24715 heartbeater.cc:347] Connected to a master server at 127.23.100.62:35229
I20260925 10:39:06.196694 24715 heartbeater.cc:464] Registering TS with master...
I20260925 10:39:06.197461 24715 heartbeater.cc:511] Master 127.23.100.62:35229 requested a full tablet report, sending...
I20260925 10:39:06.199409 24526 ts_manager.cc:194] Registered new tserver with Master: ffc4c9bfd37a4e2b8db9899e938e87bd (127.23.100.1:43993)
WARNING: no policy specified for kudu/127.23.100.2@KRBTEST.COM; defaulting to no policy
I20260925 10:39:06.200822 24526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.1@KRBTEST.COM'} at 127.23.100.1:37773
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab.
Entry for principal kudu/127.23.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab.
I20260925 10:39:06.221225 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.2:0
--local_ip_for_outbound_sockets=127.23.100.2
--webserver_interface=127.23.100.2
--webserver_port=0
--tserver_master_addrs=127.23.100.62:35229
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab
--principal=kudu/127.23.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:38145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:39:06.534368 24721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:39:06.534667 24721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:39:06.534821 24721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:39:06.541654 24721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:39:06.541791 24721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:39:06.541952 24721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.2
I20260925 10:39:06.551245 24721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:38145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab
--principal=kudu/127.23.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.2
--webserver_port=0
--tserver_master_addrs=127.23.100.62:35229
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.2
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:39:06.552387 24721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:39:06.553666 24721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:39:06.565976 24727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:06.566166 24726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:06.566635 24729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:06.566980 24721 server_base.cc:1034] running on GCE node
I20260925 10:39:06.567477 24721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:39:06.568334 24721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:39:06.569531 24721 hybrid_clock.cc:648] HybridClock initialized: now 1790332746569491 us; error 29 us; skew 500 ppm
I20260925 10:39:06.569834 24721 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:39:06 dist-test-slave-dxdd krb5kdc[24481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332746, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:39:06.572779 24721 init.cc:377] Logged in from keytab as kudu/127.23.100.2@KRBTEST.COM (short username kudu)
I20260925 10:39:06.574009 24721 webserver.cc:460] Webserver started at http://127.23.100.2:36605/ using document root <none> and password file <none>
I20260925 10:39:06.574646 24721 fs_manager.cc:359] Metadata directory not provided
I20260925 10:39:06.574739 24721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:39:06.574967 24721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:39:06.577679 24721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/ts-1/data/instance:
uuid: "a69fb7ab1c474fe684c9500164b511e1"
format_stamp: "Formatted at 2026-09-25 10:39:06 on dist-test-slave-dxdd"
server_key: "1613ac82083d4961501242b7e5ee7b28"
server_key_iv: "3dd7121e444d5c16892f2e7dd1939160"
server_key_version: "encryptionkey@0"
I20260925 10:39:06.578459 24721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/ts-1/wal/instance:
uuid: "a69fb7ab1c474fe684c9500164b511e1"
format_stamp: "Formatted at 2026-09-25 10:39:06 on dist-test-slave-dxdd"
server_key: "1613ac82083d4961501242b7e5ee7b28"
server_key_iv: "3dd7121e444d5c16892f2e7dd1939160"
server_key_version: "encryptionkey@0"
I20260925 10:39:06.584477 24721 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260925 10:39:06.588493 24736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:06.589847 24721 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:39:06.589995 24721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/ts-1/wal
uuid: "a69fb7ab1c474fe684c9500164b511e1"
format_stamp: "Formatted at 2026-09-25 10:39:06 on dist-test-slave-dxdd"
server_key: "1613ac82083d4961501242b7e5ee7b28"
server_key_iv: "3dd7121e444d5c16892f2e7dd1939160"
server_key_version: "encryptionkey@0"
I20260925 10:39:06.590186 24721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:39:06.603524 24721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:39:06.607089 24721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:39:06.607336 24721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:39:06.608150 24721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:39:06.609292 24721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:39:06.609356 24721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:06.609439 24721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:39:06.609481 24721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:06.641577 24721 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.2:39993
I20260925 10:39:06.641626 24848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.2:39993 every 8 connection(s)
I20260925 10:39:06.642797 24721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
I20260925 10:39:06.646413 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 24721
I20260925 10:39:06.646607 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/ts-1/wal/instance
I20260925 10:39:06.647187 23952 external_mini_cluster.cc:1428] Setting key 3c3986a82217634b7a38689dcfc45102
Sep 25 10:39:06 dist-test-slave-dxdd krb5kdc[24481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332746, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.2@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:39:06.664832 24717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:06.645166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:59623 (local address 127.23.100.62:35229)
0925 10:39:06.645378 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:06.645384 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:06.646653 (+ 1269us) server_negotiation.cc:407] Connection header received
0925 10:39:06.648834 (+ 2181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:06.648840 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:06.648920 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:06.648999 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:06.650858 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:06.651553 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:06.653966 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:06.654412 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:06.654556 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:06.658027 (+ 3471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:06.658037 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:06.658045 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:06.658089 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:06.661632 (+ 3543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:06.662388 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:06.662396 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:06.662402 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:06.662569 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:06.662979 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:06.662986 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:06.662993 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:06.663413 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:06.663584 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:06.664172 (+ 588us) server_negotiation.cc:299] Negotiation successful
0925 10:39:06.664426 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 10:39:06.665401 24851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:06.645668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35229 (local address 127.23.100.2:59623)
0925 10:39:06.646489 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:06.646538 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:06.648577 (+ 2039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:06.649185 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:06.649192 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:06.649870 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:06.650687 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:06.650714 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:06.651698 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:06.651706 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:06.653803 (+ 2097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:06.653841 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:06.654617 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:06.654622 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:06.654838 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:06.655505 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:06.655526 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:06.657818 (+ 2292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:06.661794 (+ 3976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:06.661803 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:06.661833 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:06.662246 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:06.662680 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:06.662683 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:06.662686 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:06.662860 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:06.663531 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:06.663536 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:06.663820 (+ 284us) client_negotiation.cc:769] Sending connection context
0925 10:39:06.664029 (+ 209us) client_negotiation.cc:240] Negotiation successful
0925 10:39:06.664371 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":293,"threads_started":1}
WARNING: no policy specified for kudu/127.23.100.3@KRBTEST.COM; defaulting to no policy
I20260925 10:39:06.666849 24849 heartbeater.cc:347] Connected to a master server at 127.23.100.62:35229
I20260925 10:39:06.667223 24849 heartbeater.cc:464] Registering TS with master...
I20260925 10:39:06.668030 24849 heartbeater.cc:511] Master 127.23.100.62:35229 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.3@KRBTEST.COM" created.
I20260925 10:39:06.669656 24526 ts_manager.cc:194] Registered new tserver with Master: a69fb7ab1c474fe684c9500164b511e1 (127.23.100.2:39993)
I20260925 10:39:06.670965 24526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.2@KRBTEST.COM'} at 127.23.100.2:59623
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab.
Entry for principal kudu/127.23.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab.
I20260925 10:39:06.688769 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.3:0
--local_ip_for_outbound_sockets=127.23.100.3
--webserver_interface=127.23.100.3
--webserver_port=0
--tserver_master_addrs=127.23.100.62:35229
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab
--principal=kudu/127.23.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:38145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:39:06.993291 24854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:39:06.993558 24854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:39:06.993736 24854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:39:07.000924 24854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:39:07.001056 24854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:39:07.001235 24854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.3
I20260925 10:39:07.010587 24854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:38145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab
--principal=kudu/127.23.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.3
--webserver_port=0
--tserver_master_addrs=127.23.100.62:35229
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.3
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:39:07.011808 24854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:39:07.013052 24854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:39:07.025588 24859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:07.025666 24862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:07.025609 24860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:07.027055 24854 server_base.cc:1034] running on GCE node
I20260925 10:39:07.027665 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:39:07.028542 24854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:39:07.029764 24854 hybrid_clock.cc:648] HybridClock initialized: now 1790332747029730 us; error 42 us; skew 500 ppm
I20260925 10:39:07.030059 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:39:07 dist-test-slave-dxdd krb5kdc[24481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332747, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:39:07.033114 24854 init.cc:377] Logged in from keytab as kudu/127.23.100.3@KRBTEST.COM (short username kudu)
I20260925 10:39:07.034489 24854 webserver.cc:460] Webserver started at http://127.23.100.3:46225/ using document root <none> and password file <none>
I20260925 10:39:07.035113 24854 fs_manager.cc:359] Metadata directory not provided
I20260925 10:39:07.035248 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:39:07.035549 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:39:07.038340 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/ts-2/data/instance:
uuid: "d7374a0f4c284eb28d9e1e530e42d6b2"
format_stamp: "Formatted at 2026-09-25 10:39:07 on dist-test-slave-dxdd"
server_key: "519bdde035b0ecf4428044c6758e7e20"
server_key_iv: "393611a728bfa5bae543a266b6e21eb6"
server_key_version: "encryptionkey@0"
I20260925 10:39:07.039099 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/ts-2/wal/instance:
uuid: "d7374a0f4c284eb28d9e1e530e42d6b2"
format_stamp: "Formatted at 2026-09-25 10:39:07 on dist-test-slave-dxdd"
server_key: "519bdde035b0ecf4428044c6758e7e20"
server_key_iv: "393611a728bfa5bae543a266b6e21eb6"
server_key_version: "encryptionkey@0"
I20260925 10:39:07.043809 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 10:39:07.051443 24869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:07.052964 24854 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.006s sys 0.001s
I20260925 10:39:07.053110 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/ts-2/wal
uuid: "d7374a0f4c284eb28d9e1e530e42d6b2"
format_stamp: "Formatted at 2026-09-25 10:39:07 on dist-test-slave-dxdd"
server_key: "519bdde035b0ecf4428044c6758e7e20"
server_key_iv: "393611a728bfa5bae543a266b6e21eb6"
server_key_version: "encryptionkey@0"
I20260925 10:39:07.053332 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:39:07.075766 24854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:39:07.079430 24854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:39:07.079689 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:39:07.080622 24854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:39:07.081863 24854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:39:07.081938 24854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:07.082088 24854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:39:07.082154 24854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:07.113798 24854 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.3:39603
I20260925 10:39:07.113852 24981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.3:39603 every 8 connection(s)
I20260925 10:39:07.115108 24854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
I20260925 10:39:07.123222 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 24854
I20260925 10:39:07.123404 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790332740062128-23952-0/minicluster-data/ts-2/wal/instance
I20260925 10:39:07.123916 23952 external_mini_cluster.cc:1428] Setting key 7bb1f7ca1f9ac6de68aa6eec5fa4540a
Sep 25 10:39:07 dist-test-slave-dxdd krb5kdc[24481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332747, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.3@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:39:07.136195 24717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:07.117643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:57103 (local address 127.23.100.62:35229)
0925 10:39:07.117885 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:07.117892 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:07.119108 (+ 1216us) server_negotiation.cc:407] Connection header received
0925 10:39:07.120899 (+ 1791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:07.120904 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:07.120969 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:07.121057 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:07.122510 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:07.123535 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:07.125623 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:07.126162 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:07.126291 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:07.129497 (+ 3206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:07.129502 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:07.129506 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:07.129537 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:07.132240 (+ 2703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:07.133016 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:07.133022 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:07.133025 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:07.133175 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:07.133809 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:07.133815 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:07.133818 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:07.134193 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:07.134399 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:07.135664 (+ 1265us) server_negotiation.cc:299] Negotiation successful
0925 10:39:07.135892 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260925 10:39:07.136780 24984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:07.118201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35229 (local address 127.23.100.3:57103)
0925 10:39:07.118952 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:07.118993 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:07.120632 (+ 1639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:07.121252 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:07.121260 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:07.121751 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:07.122295 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:07.122372 (+ 77us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:07.123699 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:07.123704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:07.125482 (+ 1778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:07.125494 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:07.126423 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:07.126428 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:07.126676 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:07.127288 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:07.127316 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:07.129292 (+ 1976us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:07.132352 (+ 3060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:07.132358 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:07.132370 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:07.132801 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:07.133380 (+ 579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:07.133385 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:07.133388 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:07.133669 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:07.134341 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:07.134350 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:07.134752 (+ 402us) client_negotiation.cc:769] Sending connection context
0925 10:39:07.135051 (+ 299us) client_negotiation.cc:240] Negotiation successful
0925 10:39:07.135626 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":266,"threads_started":1}
I20260925 10:39:07.138265 24982 heartbeater.cc:347] Connected to a master server at 127.23.100.62:35229
I20260925 10:39:07.138600 24982 heartbeater.cc:464] Registering TS with master...
I20260925 10:39:07.139334 24982 heartbeater.cc:511] Master 127.23.100.62:35229 requested a full tablet report, sending...
I20260925 10:39:07.140681 24525 ts_manager.cc:194] Registered new tserver with Master: d7374a0f4c284eb28d9e1e530e42d6b2 (127.23.100.3:39603)
I20260925 10:39:07.141497 24525 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.3@KRBTEST.COM'} at 127.23.100.3:57103
I20260925 10:39:07.150878 23952 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:39:07.168694 24717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:07.156830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43784 (local address 127.23.100.62:35229)
0925 10:39:07.157032 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:07.157041 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:07.157473 (+ 432us) server_negotiation.cc:407] Connection header received
0925 10:39:07.157616 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:07.157620 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:07.157679 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:07.157769 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:07.159218 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:07.159961 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:07.161150 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:07.161533 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:07.161620 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:07.163799 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:07.163804 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:07.163807 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:07.163842 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:07.166531 (+ 2689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:07.167074 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:07.167077 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:07.167080 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:07.167198 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:07.167489 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:07.167492 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:07.167494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:07.167771 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:07.167947 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:07.168235 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 10:39:07.168374 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 10:39:07.174688 24525 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43784:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:39:07.181207 24525 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:39:07.202078 24994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:07.192369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.2:39993 (local address 127.0.0.1:45830)
0925 10:39:07.193115 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:07.193162 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:07.193319 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:07.194171 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:07.194177 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:07.194209 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:39:07.194694 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:07.194712 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:07.196721 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:07.196725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:07.199039 (+ 2314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:07.199051 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:07.201136 (+ 2085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:07.201142 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:07.201352 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:07.201415 (+ 63us) client_negotiation.cc:769] Sending connection context
0925 10:39:07.201527 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 10:39:07.201701 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":360,"threads_started":1}
I20260925 10:39:07.202070 24995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:07.192677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45830 (local address 127.23.100.2:39993)
0925 10:39:07.193180 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:07.193193 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:07.193224 (+ 31us) server_negotiation.cc:407] Connection header received
0925 10:39:07.193601 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:07.193610 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:07.193893 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:07.194076 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:39:07.194896 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:07.196561 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:07.199185 (+ 2624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:07.200670 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:07.200823 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:07.201007 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:07.201625 (+ 618us) server_negotiation.cc:299] Negotiation successful
0925 10:39:07.201781 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":228,"threads_started":1}
I20260925 10:39:07.202800 24996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:07.193493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.3:39603 (local address 127.0.0.1:34270)
0925 10:39:07.194743 (+ 1250us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:07.194819 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:07.194920 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:07.196220 (+ 1300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:07.196225 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:07.196250 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:39:07.196597 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:07.196611 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:07.198815 (+ 2204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:07.198820 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:07.200520 (+ 1700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:07.200534 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:07.202032 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:07.202038 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:07.202196 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:07.202229 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 10:39:07.202293 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 10:39:07.202443 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1092,"thread_start_us":961,"threads_started":1}
I20260925 10:39:07.203637 24715 heartbeater.cc:503] Master 127.23.100.62:35229 was elected leader, sending a full tablet report...
I20260925 10:39:07.204042 24998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:07.194871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:43993 (local address 127.0.0.1:47326)
0925 10:39:07.195934 (+ 1063us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:07.195956 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:07.196091 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:07.197416 (+ 1325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:07.197420 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:07.197446 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:39:07.197825 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:07.197834 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:07.199354 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:07.199359 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:07.201424 (+ 2065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:07.201436 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:07.203339 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:07.203346 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:07.203510 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:07.203531 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 10:39:07.203629 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 10:39:07.203743 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":928,"thread_start_us":808,"threads_started":1}
I20260925 10:39:07.204449 24999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:07.195378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47326 (local address 127.23.100.1:43993)
0925 10:39:07.196639 (+ 1261us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:07.196646 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:07.196671 (+ 25us) server_negotiation.cc:407] Connection header received
0925 10:39:07.196760 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:07.196770 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:07.196978 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:07.197162 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:39:07.197992 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:07.199220 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:07.201575 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:07.203059 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:07.203196 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:07.203423 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:07.203675 (+ 252us) server_negotiation.cc:299] Negotiation successful
0925 10:39:07.203842 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1114,"thread_start_us":999,"threads_started":1}
I20260925 10:39:07.204706 24997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:07.193834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34270 (local address 127.23.100.3:39603)
0925 10:39:07.195525 (+ 1691us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:07.195534 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:07.195565 (+ 31us) server_negotiation.cc:407] Connection header received
0925 10:39:07.195677 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:07.195685 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:07.195973 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:07.196775 (+ 802us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:39:07.197078 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:07.198667 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:07.200693 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:07.201804 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:07.202733 (+ 929us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:07.202983 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:07.203047 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 10:39:07.203194 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1523,"thread_start_us":1376,"threads_started":1}
I20260925 10:39:07.206498 24784 tablet_service.cc:1467] Processing CreateTablet for tablet f342ff28255847b9bc40dd419e6ccd25 (DEFAULT_TABLE table=test-table [id=93044f413a0f4c6f90eefe0b9dfbbff3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:39:07.207639 24784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f342ff28255847b9bc40dd419e6ccd25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:39:07.207967 24650 tablet_service.cc:1467] Processing CreateTablet for tablet f342ff28255847b9bc40dd419e6ccd25 (DEFAULT_TABLE table=test-table [id=93044f413a0f4c6f90eefe0b9dfbbff3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:39:07.207967 24917 tablet_service.cc:1467] Processing CreateTablet for tablet f342ff28255847b9bc40dd419e6ccd25 (DEFAULT_TABLE table=test-table [id=93044f413a0f4c6f90eefe0b9dfbbff3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:39:07.209213 24650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f342ff28255847b9bc40dd419e6ccd25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:39:07.209230 24917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f342ff28255847b9bc40dd419e6ccd25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:39:07.218552 25000 tablet_bootstrap.cc:492] T f342ff28255847b9bc40dd419e6ccd25 P a69fb7ab1c474fe684c9500164b511e1: Bootstrap starting.
I20260925 10:39:07.222139 25000 tablet_bootstrap.cc:654] T f342ff28255847b9bc40dd419e6ccd25 P a69fb7ab1c474fe684c9500164b511e1: Neither blocks nor log segments found. Creating new log.
I20260925 10:39:07.222604 25001 tablet_bootstrap.cc:492] T f342ff28255847b9bc40dd419e6ccd25 P d7374a0f4c284eb28d9e1e530e42d6b2: Bootstrap starting.
I20260925 10:39:07.224048 25002 tablet_bootstrap.cc:492] T f342ff28255847b9bc40dd419e6ccd25 P ffc4c9bfd37a4e2b8db9899e938e87bd: Bootstrap starting.
I20260925 10:39:07.224620 25000 log.cc:824] T f342ff28255847b9bc40dd419e6ccd25 P a69fb7ab1c474fe684c9500164b511e1: Log is configured to *not* fsync() on all Append() calls
I20260925 10:39:07.226243 25001 tablet_bootstrap.cc:654] T f342ff28255847b9bc40dd419e6ccd25 P d7374a0f4c284eb28d9e1e530e42d6b2: Neither blocks nor log segments found. Creating new log.
I20260925 10:39:07.227460 25001 log.cc:824] T f342ff28255847b9bc40dd419e6ccd25 P d7374a0f4c284eb28d9e1e530e42d6b2: Log is configured to *not* fsync() on all Append() calls
I20260925 10:39:07.227605 25002 tablet_bootstrap.cc:654] T f342ff28255847b9bc40dd419e6ccd25 P ffc4c9bfd37a4e2b8db9899e938e87bd: Neither blocks nor log segments found. Creating new log.
I20260925 10:39:07.228657 25002 log.cc:824] T f342ff28255847b9bc40dd419e6ccd25 P ffc4c9bfd37a4e2b8db9899e938e87bd: Log is configured to *not* fsync() on all Append() calls
I20260925 10:39:07.228829 25000 tablet_bootstrap.cc:492] T f342ff28255847b9bc40dd419e6ccd25 P a69fb7ab1c474fe684c9500164b511e1: No bootstrap required, opened a new log
I20260925 10:39:07.229074 25000 ts_tablet_manager.cc:1397] T f342ff28255847b9bc40dd419e6ccd25 P a69fb7ab1c474fe684c9500164b511e1: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.002s
I20260925 10:39:07.230247 25001 tablet_bootstrap.cc:492] T f342ff28255847b9bc40dd419e6ccd25 P d7374a0f4c284eb28d9e1e530e42d6b2: No bootstrap required, opened a new log
I20260925 10:39:07.230556 25001 ts_tablet_manager.cc:1397] T f342ff28255847b9bc40dd419e6ccd25 P d7374a0f4c284eb28d9e1e530e42d6b2: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.005s
I20260925 10:39:07.231474 25002 tablet_bootstrap.cc:492] T f342ff28255847b9bc40dd419e6ccd25 P ffc4c9bfd37a4e2b8db9899e938e87bd: No bootstrap required, opened a new log
I20260925 10:39:07.231716 25002 ts_tablet_manager.cc:1397] T f342ff28255847b9bc40dd419e6ccd25 P ffc4c9bfd37a4e2b8db9899e938e87bd: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.004s
I20260925 10:39:07.237006 25000 raft_consensus.cc:348] T f342ff28255847b9bc40dd419e6ccd25 P a69fb7ab1c474fe684c9500164b511e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a69fb7ab1c474fe684c9500164b511e1" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 39993 } } peers { permanent_uuid: "d7374a0f4c284eb28d9e1e530e42d6b2" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 39603 } } peers { permanent_uuid: "ffc4c9bfd37a4e2b8db9899e938e87bd" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 43993 } }
I20260925 10:39:07.237404 25000 raft_consensus.cc:374] T f342ff28255847b9bc40dd419e6ccd25 P a69fb7ab1c474fe684c9500164b511e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:39:07.237514 25000 raft_consensus.cc:729] T f342ff28255847b9bc40dd419e6ccd25 P a69fb7ab1c474fe684c9500164b511e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a69fb7ab1c474fe684c9500164b511e1, State: Initialized, Role: FOLLOWER
I20260925 10:39:07.238109 25000 consensus_queue.cc:260] T f342ff28255847b9bc40dd419e6ccd25 P a69fb7ab1c474fe684c9500164b511e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a69fb7ab1c474fe684c9500164b511e1" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 39993 } } peers { permanent_uuid: "d7374a0f4c284eb28d9e1e530e42d6b2" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 39603 } } peers { permanent_uuid: "ffc4c9bfd37a4e2b8db9899e938e87bd" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 43993 } }
I20260925 10:39:07.239284 24849 heartbeater.cc:503] Master 127.23.100.62:35229 was elected leader, sending a full tablet report...
I20260925 10:39:07.239789 25000 ts_tablet_manager.cc:1428] T f342ff28255847b9bc40dd419e6ccd25 P a69fb7ab1c474fe684c9500164b511e1: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
I20260925 10:39:07.241103 25001 raft_consensus.cc:348] T f342ff28255847b9bc40dd419e6ccd25 P d7374a0f4c284eb28d9e1e530e42d6b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a69fb7ab1c474fe684c9500164b511e1" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 39993 } } peers { permanent_uuid: "d7374a0f4c284eb28d9e1e530e42d6b2" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 39603 } } peers { permanent_uuid: "ffc4c9bfd37a4e2b8db9899e938e87bd" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 43993 } }
I20260925 10:39:07.241595 25001 raft_consensus.cc:374] T f342ff28255847b9bc40dd419e6ccd25 P d7374a0f4c284eb28d9e1e530e42d6b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:39:07.241722 25001 raft_consensus.cc:729] T f342ff28255847b9bc40dd419e6ccd25 P d7374a0f4c284eb28d9e1e530e42d6b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7374a0f4c284eb28d9e1e530e42d6b2, State: Initialized, Role: FOLLOWER
I20260925 10:39:07.242779 25001 consensus_queue.cc:260] T f342ff28255847b9bc40dd419e6ccd25 P d7374a0f4c284eb28d9e1e530e42d6b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a69fb7ab1c474fe684c9500164b511e1" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 39993 } } peers { permanent_uuid: "d7374a0f4c284eb28d9e1e530e42d6b2" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 39603 } } peers { permanent_uuid: "ffc4c9bfd37a4e2b8db9899e938e87bd" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 43993 } }
I20260925 10:39:07.243063 25002 raft_consensus.cc:348] T f342ff28255847b9bc40dd419e6ccd25 P ffc4c9bfd37a4e2b8db9899e938e87bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a69fb7ab1c474fe684c9500164b511e1" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 39993 } } peers { permanent_uuid: "d7374a0f4c284eb28d9e1e530e42d6b2" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 39603 } } peers { permanent_uuid: "ffc4c9bfd37a4e2b8db9899e938e87bd" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 43993 } }
I20260925 10:39:07.243582 25002 raft_consensus.cc:374] T f342ff28255847b9bc40dd419e6ccd25 P ffc4c9bfd37a4e2b8db9899e938e87bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:39:07.243722 25002 raft_consensus.cc:729] T f342ff28255847b9bc40dd419e6ccd25 P ffc4c9bfd37a4e2b8db9899e938e87bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffc4c9bfd37a4e2b8db9899e938e87bd, State: Initialized, Role: FOLLOWER
I20260925 10:39:07.243971 24982 heartbeater.cc:503] Master 127.23.100.62:35229 was elected leader, sending a full tablet report...
I20260925 10:39:07.244318 25002 consensus_queue.cc:260] T f342ff28255847b9bc40dd419e6ccd25 P ffc4c9bfd37a4e2b8db9899e938e87bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a69fb7ab1c474fe684c9500164b511e1" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 39993 } } peers { permanent_uuid: "d7374a0f4c284eb28d9e1e530e42d6b2" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 39603 } } peers { permanent_uuid: "ffc4c9bfd37a4e2b8db9899e938e87bd" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 43993 } }
I20260925 10:39:07.244637 25001 ts_tablet_manager.cc:1428] T f342ff28255847b9bc40dd419e6ccd25 P d7374a0f4c284eb28d9e1e530e42d6b2: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260925 10:39:07.245998 25002 ts_tablet_manager.cc:1428] T f342ff28255847b9bc40dd419e6ccd25 P ffc4c9bfd37a4e2b8db9899e938e87bd: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
W20260925 10:39:07.366760 24983 tablet.cc:2369] T f342ff28255847b9bc40dd419e6ccd25 P d7374a0f4c284eb28d9e1e530e42d6b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:39:07.394743 24850 tablet.cc:2369] T f342ff28255847b9bc40dd419e6ccd25 P a69fb7ab1c474fe684c9500164b511e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:39:07.426908 24716 tablet.cc:2369] T f342ff28255847b9bc40dd419e6ccd25 P ffc4c9bfd37a4e2b8db9899e938e87bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:39:07.464644 25007 raft_consensus.cc:482] T f342ff28255847b9bc40dd419e6ccd25 P d7374a0f4c284eb28d9e1e530e42d6b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:39:07.464972 25007 raft_consensus.cc:504] T f342ff28255847b9bc40dd419e6ccd25 P d7374a0f4c284eb28d9e1e530e42d6b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a69fb7ab1c474fe684c9500164b511e1" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 39993 } } peers { permanent_uuid: "d7374a0f4c284eb28d9e1e530e42d6b2" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 39603 } } peers { permanent_uuid: "ffc4c9bfd37a4e2b8db9899e938e87bd" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 43993 } }
I20260925 10:39:07.466691 25007 leader_election.cc:288] T f342ff28255847b9bc40dd419e6ccd25 P d7374a0f4c284eb28d9e1e530e42d6b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a69fb7ab1c474fe684c9500164b511e1 (127.23.100.2:39993), ffc4c9bfd37a4e2b8db9899e938e87bd (127.23.100.1:43993)
I20260925 10:39:07.472218 24984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:07.467083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.2:39993 (local address 127.23.100.3:45117)
0925 10:39:07.467296 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:07.467313 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:07.467437 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:07.467989 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:07.467993 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:07.468025 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:39:07.468189 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:07.468196 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:07.469204 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:07.469213 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:07.470478 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:07.470492 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:07.471517 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:07.471523 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:07.471696 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:07.471719 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 10:39:07.471818 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 10:39:07.471942 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":2}
I20260925 10:39:07.472218 24995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:07.467164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:45117 (local address 127.23.100.2:39993)
0925 10:39:07.467413 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:07.467420 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:07.467441 (+ 21us) server_negotiation.cc:407] Connection header received
0925 10:39:07.467592 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:07.467599 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:07.467683 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:07.467788 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:39:07.468330 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:07.469076 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:07.470618 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:07.471355 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:07.471530 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:07.471600 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:07.471849 (+ 249us) server_negotiation.cc:299] Negotiation successful
0925 10:39:07.471973 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 10:39:07.473104 24999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:07.467773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:60641 (local address 127.23.100.1:43993)
0925 10:39:07.468006 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:07.468012 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:07.468284 (+ 272us) server_negotiation.cc:407] Connection header received
0925 10:39:07.468475 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:07.468480 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:07.468543 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:07.468629 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:39:07.469100 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:07.469760 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:07.471421 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:07.472199 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:07.472374 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:07.472444 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:07.472726 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 10:39:07.472852 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260925 10:39:07.473189 25009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:07.467518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:43993 (local address 127.23.100.3:60641)
0925 10:39:07.468183 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:07.468200 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:07.468340 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:07.468642 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:07.468646 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:07.468666 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:39:07.468963 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:07.468970 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:07.469898 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:07.469903 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:07.471247 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:07.471258 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:07.472399 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:07.472405 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:07.472564 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:07.472594 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 10:39:07.472696 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 10:39:07.472812 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"mutex_wait_us":29,"thread_start_us":360,"threads_started":1}
I20260925 10:39:07.473326 24804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f342ff28255847b9bc40dd419e6ccd25" candidate_uuid: "d7374a0f4c284eb28d9e1e530e42d6b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a69fb7ab1c474fe684c9500164b511e1" is_pre_election: true
I20260925 10:39:07.473816 24804 raft_consensus.cc:2454] T f342ff28255847b9bc40dd419e6ccd25 P a69fb7ab1c474fe684c9500164b511e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7374a0f4c284eb28d9e1e530e42d6b2 in term 0.
I20260925 10:39:07.474148 24670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f342ff28255847b9bc40dd419e6ccd25" candidate_uuid: "d7374a0f4c284eb28d9e1e530e42d6b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffc4c9bfd37a4e2b8db9899e938e87bd" is_pre_election: true
I20260925 10:39:07.474522 24871 leader_election.cc:302] T f342ff28255847b9bc40dd419e6ccd25 P d7374a0f4c284eb28d9e1e530e42d6b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a69fb7ab1c474fe684c9500164b511e1, d7374a0f4c284eb28d9e1e530e42d6b2; no voters:
I20260925 10:39:07.474718 24670 raft_consensus.cc:2454] T f342ff28255847b9bc40dd419e6ccd25 P ffc4c9bfd37a4e2b8db9899e938e87bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7374a0f4c284eb28d9e1e530e42d6b2 in term 0.
I20260925 10:39:07.474933 25007 raft_consensus.cc:2781] T f342ff28255847b9bc40dd419e6ccd25 P d7374a0f4c284eb28d9e1e530e42d6b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:39:07.475080 25007 raft_consensus.cc:482] T f342ff28255847b9bc40dd419e6ccd25 P d7374a0f4c284eb28d9e1e530e42d6b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:39:07.475196 25007 raft_consensus.cc:3037] T f342ff28255847b9bc40dd419e6ccd25 P d7374a0f4c284eb28d9e1e530e42d6b2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:39:07.477355 25007 raft_consensus.cc:504] T f342ff28255847b9bc40dd419e6ccd25 P d7374a0f4c284eb28d9e1e530e42d6b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a69fb7ab1c474fe684c9500164b511e1" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 39993 } } peers { permanent_uuid: "d7374a0f4c284eb28d9e1e530e42d6b2" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 39603 } } peers { permanent_uuid: "ffc4c9bfd37a4e2b8db9899e938e87bd" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 43993 } }
I20260925 10:39:07.477895 25007 leader_election.cc:288] T f342ff28255847b9bc40dd419e6ccd25 P d7374a0f4c284eb28d9e1e530e42d6b2 [CANDIDATE]: Term 1 election: Requested vote from peers a69fb7ab1c474fe684c9500164b511e1 (127.23.100.2:39993), ffc4c9bfd37a4e2b8db9899e938e87bd (127.23.100.1:43993)
I20260925 10:39:07.478272 24804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f342ff28255847b9bc40dd419e6ccd25" candidate_uuid: "d7374a0f4c284eb28d9e1e530e42d6b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a69fb7ab1c474fe684c9500164b511e1"
I20260925 10:39:07.478376 24670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f342ff28255847b9bc40dd419e6ccd25" candidate_uuid: "d7374a0f4c284eb28d9e1e530e42d6b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffc4c9bfd37a4e2b8db9899e938e87bd"
I20260925 10:39:07.478498 24804 raft_consensus.cc:3037] T f342ff28255847b9bc40dd419e6ccd25 P a69fb7ab1c474fe684c9500164b511e1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:39:07.478576 24670 raft_consensus.cc:3037] T f342ff28255847b9bc40dd419e6ccd25 P ffc4c9bfd37a4e2b8db9899e938e87bd [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:39:07.480621 24804 raft_consensus.cc:2454] T f342ff28255847b9bc40dd419e6ccd25 P a69fb7ab1c474fe684c9500164b511e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7374a0f4c284eb28d9e1e530e42d6b2 in term 1.
I20260925 10:39:07.480656 24670 raft_consensus.cc:2454] T f342ff28255847b9bc40dd419e6ccd25 P ffc4c9bfd37a4e2b8db9899e938e87bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7374a0f4c284eb28d9e1e530e42d6b2 in term 1.
I20260925 10:39:07.480968 24871 leader_election.cc:302] T f342ff28255847b9bc40dd419e6ccd25 P d7374a0f4c284eb28d9e1e530e42d6b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a69fb7ab1c474fe684c9500164b511e1, d7374a0f4c284eb28d9e1e530e42d6b2; no voters:
I20260925 10:39:07.481246 25007 raft_consensus.cc:2781] T f342ff28255847b9bc40dd419e6ccd25 P d7374a0f4c284eb28d9e1e530e42d6b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:39:07.481666 25007 raft_consensus.cc:686] T f342ff28255847b9bc40dd419e6ccd25 P d7374a0f4c284eb28d9e1e530e42d6b2 [term 1 LEADER]: Becoming Leader. State: Replica: d7374a0f4c284eb28d9e1e530e42d6b2, State: Running, Role: LEADER
I20260925 10:39:07.482115 25007 consensus_queue.cc:237] T f342ff28255847b9bc40dd419e6ccd25 P d7374a0f4c284eb28d9e1e530e42d6b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a69fb7ab1c474fe684c9500164b511e1" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 39993 } } peers { permanent_uuid: "d7374a0f4c284eb28d9e1e530e42d6b2" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 39603 } } peers { permanent_uuid: "ffc4c9bfd37a4e2b8db9899e938e87bd" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 43993 } }
I20260925 10:39:07.486617 24526 catalog_manager.cc:5495] T f342ff28255847b9bc40dd419e6ccd25 P d7374a0f4c284eb28d9e1e530e42d6b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d7374a0f4c284eb28d9e1e530e42d6b2 (127.23.100.3). New cstate: current_term: 1 leader_uuid: "d7374a0f4c284eb28d9e1e530e42d6b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a69fb7ab1c474fe684c9500164b511e1" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 39993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7374a0f4c284eb28d9e1e530e42d6b2" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 39603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ffc4c9bfd37a4e2b8db9899e938e87bd" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 43993 } health_report { overall_health: UNKNOWN } } }
Sep 25 10:39:07 dist-test-slave-dxdd krb5kdc[24481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.1@KRBTEST.COM
I20260925 10:39:07.572192 24999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:07.557810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47338 (local address 127.23.100.1:43993)
0925 10:39:07.557990 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:07.558012 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:07.558603 (+ 591us) server_negotiation.cc:407] Connection header received
0925 10:39:07.558804 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:07.558810 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:07.558881 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:07.558984 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:07.560627 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:07.561366 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:07.562698 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:07.563100 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:07.563246 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:07.567202 (+ 3956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:07.567219 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:07.567236 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:07.567270 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:07.569883 (+ 2613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:07.570569 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:07.570577 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:07.570580 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:07.570709 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:07.571029 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:07.571032 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:07.571034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:07.571368 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:07.571580 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:07.571767 (+ 187us) server_negotiation.cc:299] Negotiation successful
0925 10:39:07.571932 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Sep 25 10:39:07 dist-test-slave-dxdd krb5kdc[24481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332747, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 10:39:07 dist-test-slave-dxdd krb5kdc[24481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332747, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.100.1@KRBTEST.COM
I20260925 10:39:07.617748 24999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:07.604609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47342 (local address 127.23.100.1:43993)
0925 10:39:07.604867 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:07.604874 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:07.605096 (+ 222us) server_negotiation.cc:407] Connection header received
0925 10:39:07.605322 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:07.605328 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:07.605401 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:07.605491 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:07.606788 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:07.607511 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:07.608679 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:07.609061 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:07.609146 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:07.612733 (+ 3587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:07.612738 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:07.612741 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:07.612770 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:07.615309 (+ 2539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:07.615881 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:07.615889 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:07.615892 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:07.616018 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:07.616393 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:07.616399 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:07.616402 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:07.616656 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:07.616924 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:07.617173 (+ 249us) server_negotiation.cc:299] Negotiation successful
0925 10:39:07.617395 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
Sep 25 10:39:07 dist-test-slave-dxdd krb5kdc[24481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 10:39:07 dist-test-slave-dxdd krb5kdc[24481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.1@KRBTEST.COM
I20260925 10:39:07.663511 24999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:07.650592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47358 (local address 127.23.100.1:43993)
0925 10:39:07.650802 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:07.650817 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:07.650979 (+ 162us) server_negotiation.cc:407] Connection header received
0925 10:39:07.651179 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:07.651185 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:07.651259 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:07.651366 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:07.652786 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:07.653573 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:07.654854 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:07.655250 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:07.655386 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:07.659025 (+ 3639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:07.659029 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:07.659032 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:07.659063 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:07.661413 (+ 2350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:07.661967 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:07.661971 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:07.661973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:07.662081 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:07.662431 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:07.662434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:07.662436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:07.662710 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:07.662914 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:07.663119 (+ 205us) server_negotiation.cc:299] Negotiation successful
0925 10:39:07.663279 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260925 10:39:07.665138 24630 tablet_service.cc:2908] Rejecting scan request for tablet f342ff28255847b9bc40dd419e6ccd25: Uninitialized: clean time has not yet been initialized
I20260925 10:39:07.668056 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 24587
I20260925 10:39:07.680187 24999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:07.669080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47374 (local address 127.23.100.1:43993)
0925 10:39:07.669311 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:07.669318 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:07.669543 (+ 225us) server_negotiation.cc:407] Connection header received
0925 10:39:07.669719 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:07.669728 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:07.669792 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:07.669910 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:07.671076 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:07.671717 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:07.672867 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:07.673268 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:07.673399 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:07.675472 (+ 2073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:07.675477 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:07.675480 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:07.675507 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:07.677713 (+ 2206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:07.678356 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:07.678362 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:07.678365 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:07.678478 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:07.678826 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:07.678832 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:07.678836 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:07.679104 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:07.679403 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:07.679663 (+ 260us) server_negotiation.cc:299] Negotiation successful
0925 10:39:07.679894 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 10:39:07.680778 24710 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:39:07.768666 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 24587
I20260925 10:39:07.785485 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 24721
Sep 25 10:39:07 dist-test-slave-dxdd krb5kdc[24481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.2@KRBTEST.COM
I20260925 10:39:07.799991 24995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:07.786173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45846 (local address 127.23.100.2:39993)
0925 10:39:07.786364 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:07.786368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:07.786386 (+ 18us) server_negotiation.cc:407] Connection header received
0925 10:39:07.786542 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:07.786546 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:07.786612 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:07.786694 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:07.788138 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:07.788864 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:07.790072 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:07.790598 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:07.790685 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:07.794284 (+ 3599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:07.794298 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:07.794353 (+ 55us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:07.794402 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:07.797321 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:07.797941 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:07.797948 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:07.797950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:07.798088 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:07.798521 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:07.798528 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:07.798531 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:07.798880 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:07.799182 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:07.799453 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 10:39:07.799668 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 10:39:07.800559 24844 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:39:07.886257 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 24721
I20260925 10:39:07.903079 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 24854
Sep 25 10:39:07 dist-test-slave-dxdd krb5kdc[24481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.3@KRBTEST.COM
I20260925 10:39:07.917166 25025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:07.903820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34276 (local address 127.23.100.3:39603)
0925 10:39:07.904228 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:07.904233 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:07.904250 (+ 17us) server_negotiation.cc:407] Connection header received
0925 10:39:07.904302 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:07.904305 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:07.904366 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:07.904453 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:07.905609 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:07.906267 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:07.907550 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:07.907943 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:07.908028 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:07.911843 (+ 3815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:07.911857 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:07.911872 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:07.911908 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:07.914563 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:07.915188 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:07.915196 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:07.915198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:07.915337 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:07.915681 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:07.915687 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:07.915691 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:07.916008 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:07.916292 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:07.916572 (+ 280us) server_negotiation.cc:299] Negotiation successful
0925 10:39:07.916801 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":208,"threads_started":1}
I20260925 10:39:07.917809 24977 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:39:08.006933 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 24854
I20260925 10:39:08.023156 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 24494
I20260925 10:39:08.024328 24556 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:39:08.129688 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 24494
2026-09-25T10:39:08Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4203 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:39:08 dist-test-slave-dxdd krb5kdc[25030](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:39:08 dist-test-slave-dxdd krb5kdc[25030](info): set up 2 sockets
Sep 25 10:39:08 dist-test-slave-dxdd krb5kdc[25030](info): commencing operation
krb5kdc: starting...
W20260925 10:39:09.206243 23952 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Sep 25 10:39:09 dist-test-slave-dxdd krb5kdc[25030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:39:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:39:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
I20260925 10:39:09.377171 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:37929
--webserver_interface=127.23.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:44945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.62:37929
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:39:09.683501 25043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:39:09.683794 25043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:39:09.683962 25043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:39:09.690894 25043 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:39:09.691006 25043 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 10:39:09.691092 25043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:39:09.691186 25043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:39:09.691267 25043 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:39:09.691356 25043 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:39:09.700629 25043 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:44945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.62:37929
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:37929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:39:09.701763 25043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:39:09.703083 25043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:39:09.714102 25048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:09.714129 25049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:09.714206 25051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:09.715246 25043 server_base.cc:1034] running on GCE node
I20260925 10:39:09.715932 25043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:39:09.717177 25043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:39:09.718411 25043 hybrid_clock.cc:648] HybridClock initialized: now 1790332749718347 us; error 41 us; skew 500 ppm
I20260925 10:39:09.718720 25043 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:39:09 dist-test-slave-dxdd krb5kdc[25030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332749, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:39:09.721979 25043 init.cc:377] Logged in from keytab as kudu/127.23.100.62@KRBTEST.COM (short username kudu)
I20260925 10:39:09.723455 25043 webserver.cc:460] Webserver started at http://127.23.100.62:40671/ using document root <none> and password file <none>
I20260925 10:39:09.724126 25043 fs_manager.cc:359] Metadata directory not provided
I20260925 10:39:09.724243 25043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:39:09.724557 25043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:39:09.727366 25043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/master-0/data/instance:
uuid: "17626f350f1e4739a7e884a2be1a8823"
format_stamp: "Formatted at 2026-09-25 10:39:09 on dist-test-slave-dxdd"
server_key: "f6f12d63d6def2c85f16168acd3c97f0"
server_key_iv: "26f74b65a8113b56cad0cb83616969f9"
server_key_version: "encryptionkey@0"
I20260925 10:39:09.728104 25043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/master-0/wal/instance:
uuid: "17626f350f1e4739a7e884a2be1a8823"
format_stamp: "Formatted at 2026-09-25 10:39:09 on dist-test-slave-dxdd"
server_key: "f6f12d63d6def2c85f16168acd3c97f0"
server_key_iv: "26f74b65a8113b56cad0cb83616969f9"
server_key_version: "encryptionkey@0"
I20260925 10:39:09.733095 25043 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 10:39:09.736248 25058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:09.737473 25043 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 10:39:09.737643 25043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/master-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/master-0/wal
uuid: "17626f350f1e4739a7e884a2be1a8823"
format_stamp: "Formatted at 2026-09-25 10:39:09 on dist-test-slave-dxdd"
server_key: "f6f12d63d6def2c85f16168acd3c97f0"
server_key_iv: "26f74b65a8113b56cad0cb83616969f9"
server_key_version: "encryptionkey@0"
I20260925 10:39:09.737839 25043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:39:09.752295 25043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:39:09.755928 25043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:39:09.756170 25043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:39:09.779294 25043 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.62:37929
I20260925 10:39:09.779316 25119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.62:37929 every 8 connection(s)
I20260925 10:39:09.780606 25043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
I20260925 10:39:09.782235 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 25043
I20260925 10:39:09.782464 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/master-0/wal/instance
I20260925 10:39:09.783083 23952 external_mini_cluster.cc:1428] Setting key dcdb0749fcf4d8e2753c3ca0e716bdda
I20260925 10:39:09.784431 25120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 10:39:09 dist-test-slave-dxdd krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:39:09.797870 25120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17626f350f1e4739a7e884a2be1a8823: Bootstrap starting.
I20260925 10:39:09.801066 25120 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17626f350f1e4739a7e884a2be1a8823: Neither blocks nor log segments found. Creating new log.
I20260925 10:39:09.802035 25120 log.cc:824] T 00000000000000000000000000000000 P 17626f350f1e4739a7e884a2be1a8823: Log is configured to *not* fsync() on all Append() calls
I20260925 10:39:09.804836 25120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17626f350f1e4739a7e884a2be1a8823: No bootstrap required, opened a new log
I20260925 10:39:09.805562 25123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:09.784852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56498 (local address 127.23.100.62:37929)
0925 10:39:09.785570 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:09.785584 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:09.785631 (+ 47us) server_negotiation.cc:407] Connection header received
0925 10:39:09.787654 (+ 2023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:09.787680 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:09.788166 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:09.788589 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:09.789879 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:09.792053 (+ 2174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:09.793210 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:09.794219 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:09.794501 (+ 282us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:09.798173 (+ 3672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:09.798181 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:09.798193 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:09.798229 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:09.801043 (+ 2814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:09.801664 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:09.801676 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:09.801681 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:09.801833 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:09.802209 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:09.802215 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:09.802218 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:09.802924 (+ 706us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:09.804059 (+ 1135us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:09.804266 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 10:39:09.804556 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":294,"threads_started":1}
I20260925 10:39:09.812748 25120 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17626f350f1e4739a7e884a2be1a8823 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17626f350f1e4739a7e884a2be1a8823" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 37929 } }
I20260925 10:39:09.813040 25120 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17626f350f1e4739a7e884a2be1a8823 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:39:09.813124 25120 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17626f350f1e4739a7e884a2be1a8823 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17626f350f1e4739a7e884a2be1a8823, State: Initialized, Role: FOLLOWER
I20260925 10:39:09.813694 25120 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17626f350f1e4739a7e884a2be1a8823 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17626f350f1e4739a7e884a2be1a8823" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 37929 } }
I20260925 10:39:09.813911 25120 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17626f350f1e4739a7e884a2be1a8823 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:39:09.814054 25120 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17626f350f1e4739a7e884a2be1a8823 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:39:09.814217 25120 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17626f350f1e4739a7e884a2be1a8823 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:39:09.816567 25120 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17626f350f1e4739a7e884a2be1a8823 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17626f350f1e4739a7e884a2be1a8823" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 37929 } }
I20260925 10:39:09.817032 25120 leader_election.cc:302] T 00000000000000000000000000000000 P 17626f350f1e4739a7e884a2be1a8823 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17626f350f1e4739a7e884a2be1a8823; no voters:
I20260925 10:39:09.817586 25120 leader_election.cc:288] T 00000000000000000000000000000000 P 17626f350f1e4739a7e884a2be1a8823 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:39:09.817711 25125 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17626f350f1e4739a7e884a2be1a8823 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:39:09.818424 25125 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17626f350f1e4739a7e884a2be1a8823 [term 1 LEADER]: Becoming Leader. State: Replica: 17626f350f1e4739a7e884a2be1a8823, State: Running, Role: LEADER
I20260925 10:39:09.818974 25125 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17626f350f1e4739a7e884a2be1a8823 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17626f350f1e4739a7e884a2be1a8823" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 37929 } }
I20260925 10:39:09.819579 25120 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17626f350f1e4739a7e884a2be1a8823 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:39:09.821550 25127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17626f350f1e4739a7e884a2be1a8823 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17626f350f1e4739a7e884a2be1a8823. Latest consensus state: current_term: 1 leader_uuid: "17626f350f1e4739a7e884a2be1a8823" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17626f350f1e4739a7e884a2be1a8823" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 37929 } } }
I20260925 10:39:09.821931 25127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17626f350f1e4739a7e884a2be1a8823 [sys.catalog]: This master's current role is: LEADER
I20260925 10:39:09.821524 25126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17626f350f1e4739a7e884a2be1a8823 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17626f350f1e4739a7e884a2be1a8823" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17626f350f1e4739a7e884a2be1a8823" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 37929 } } }
I20260925 10:39:09.822403 25126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17626f350f1e4739a7e884a2be1a8823 [sys.catalog]: This master's current role is: LEADER
I20260925 10:39:09.824337 25130 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:39:09.829835 25130 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:39:09.837642 25130 catalog_manager.cc:1347] Generated new cluster ID: c4048086da8949dfb1182d0308a1b5d5
I20260925 10:39:09.837745 25130 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:39:09.865424 25130 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:39:09.866791 25130 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:39:09.885228 25130 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17626f350f1e4739a7e884a2be1a8823: Generated new TSK 0
I20260925 10:39:09.886009 25130 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab.
Entry for principal kudu/127.23.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab.
I20260925 10:39:09.937551 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.1:0
--local_ip_for_outbound_sockets=127.23.100.1
--webserver_interface=127.23.100.1
--webserver_port=0
--tserver_master_addrs=127.23.100.62:37929
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab
--principal=kudu/127.23.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:44945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:39:10.254514 25146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:39:10.254791 25146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:39:10.254944 25146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:39:10.261924 25146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:39:10.262048 25146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:39:10.262183 25146 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 10:39:10.262326 25146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.1
I20260925 10:39:10.271482 25146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:44945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab
--principal=kudu/127.23.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.100.62:37929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.1
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:39:10.272662 25146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:39:10.273911 25146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:39:10.286579 25152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:10.286725 25154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:10.286691 25151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:10.287262 25146 server_base.cc:1034] running on GCE node
I20260925 10:39:10.287876 25146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:39:10.288714 25146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:39:10.289947 25146 hybrid_clock.cc:648] HybridClock initialized: now 1790332750289871 us; error 52 us; skew 500 ppm
I20260925 10:39:10.290225 25146 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:39:10 dist-test-slave-dxdd krb5kdc[25030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332750, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:39:10.293188 25146 init.cc:377] Logged in from keytab as kudu/127.23.100.1@KRBTEST.COM (short username kudu)
I20260925 10:39:10.294662 25146 webserver.cc:460] Webserver started at http://127.23.100.1:43895/ using document root <none> and password file <none>
I20260925 10:39:10.295271 25146 fs_manager.cc:359] Metadata directory not provided
I20260925 10:39:10.295353 25146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:39:10.295612 25146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:39:10.298225 25146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/ts-0/data/instance:
uuid: "0311d5f20f3f449db871beefa5ae2330"
format_stamp: "Formatted at 2026-09-25 10:39:10 on dist-test-slave-dxdd"
server_key: "67ed81c018de3a3613802aeb64cc285f"
server_key_iv: "1e3682a6705df367cd49beb04ed0cf56"
server_key_version: "encryptionkey@0"
I20260925 10:39:10.298990 25146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/ts-0/wal/instance:
uuid: "0311d5f20f3f449db871beefa5ae2330"
format_stamp: "Formatted at 2026-09-25 10:39:10 on dist-test-slave-dxdd"
server_key: "67ed81c018de3a3613802aeb64cc285f"
server_key_iv: "1e3682a6705df367cd49beb04ed0cf56"
server_key_version: "encryptionkey@0"
I20260925 10:39:10.303701 25146 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260925 10:39:10.307088 25161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:10.308245 25146 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.004s
I20260925 10:39:10.308372 25146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/ts-0/wal
uuid: "0311d5f20f3f449db871beefa5ae2330"
format_stamp: "Formatted at 2026-09-25 10:39:10 on dist-test-slave-dxdd"
server_key: "67ed81c018de3a3613802aeb64cc285f"
server_key_iv: "1e3682a6705df367cd49beb04ed0cf56"
server_key_version: "encryptionkey@0"
I20260925 10:39:10.308535 25146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:39:10.326683 25146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:39:10.330286 25146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:39:10.330576 25146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:39:10.332975 25146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:39:10.333050 25146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:10.333223 25146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:39:10.333277 25146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 10:39:10 dist-test-slave-dxdd krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332750, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.1@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:39:10.355814 25170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:10.333632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:58979 (local address 127.23.100.62:37929)
0925 10:39:10.334116 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:10.334123 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:10.335264 (+ 1141us) server_negotiation.cc:407] Connection header received
0925 10:39:10.337408 (+ 2144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:10.337414 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:10.337495 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:10.337615 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:10.339743 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:10.341058 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:10.343617 (+ 2559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:10.344229 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:10.344377 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:10.348851 (+ 4474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:10.348855 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:10.348858 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:10.348889 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:10.351780 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:10.352718 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:10.352724 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:10.352728 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:10.352848 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:10.353457 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:10.353465 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:10.353471 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:10.353945 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:10.354156 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:10.354980 (+ 824us) server_negotiation.cc:299] Negotiation successful
0925 10:39:10.355285 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":248,"threads_started":1}
I20260925 10:39:10.356279 25171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:10.334148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:37929 (local address 127.23.100.1:58979)
0925 10:39:10.335094 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:10.335142 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:10.337141 (+ 1999us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:10.337875 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:10.337890 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:10.338729 (+ 839us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:10.339533 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:10.339572 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:10.341236 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:10.341242 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:10.343472 (+ 2230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:10.343483 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:10.344705 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:10.344711 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:10.344944 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:10.345745 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:10.345781 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:10.348626 (+ 2845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:10.351922 (+ 3296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:10.351932 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:10.351946 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:10.352594 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:10.352965 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:10.352969 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:10.352973 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:10.353294 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:10.354144 (+ 850us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:10.354157 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:10.354586 (+ 429us) client_negotiation.cc:769] Sending connection context
0925 10:39:10.354809 (+ 223us) client_negotiation.cc:240] Negotiation successful
0925 10:39:10.355147 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"thread_start_us":463,"threads_started":1}
I20260925 10:39:10.378276 25240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:10.363775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:37929 (local address 127.23.100.1:39267)
0925 10:39:10.364544 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:10.364580 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:10.364692 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:10.365067 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:10.365073 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:10.365456 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:10.365738 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:10.365746 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:10.367333 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:10.367338 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:10.368941 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:10.368952 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:10.370064 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:10.370069 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:10.370339 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:10.371006 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:10.371030 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:10.372016 (+ 986us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:10.374914 (+ 2898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:10.374922 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:10.374927 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:10.375578 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:10.375980 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:10.375986 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:10.375990 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:10.376166 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:10.376890 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:10.376898 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:10.377162 (+ 264us) client_negotiation.cc:769] Sending connection context
0925 10:39:10.377369 (+ 207us) client_negotiation.cc:240] Negotiation successful
0925 10:39:10.377676 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":409,"threads_started":1}
I20260925 10:39:10.378976 25170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:10.363590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:39267 (local address 127.23.100.62:37929)
0925 10:39:10.363786 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:10.363794 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:10.364674 (+ 880us) server_negotiation.cc:407] Connection header received
0925 10:39:10.364804 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:10.364812 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:10.364909 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:10.365001 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:10.365881 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:10.367181 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:10.369163 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:10.369729 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:10.369876 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:10.372179 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:10.372184 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:10.372189 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:10.372226 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:10.374774 (+ 2548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:10.375718 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:10.375722 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:10.375724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:10.375839 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:10.376314 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:10.376320 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:10.376325 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:10.376689 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:10.376835 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:10.378344 (+ 1509us) server_negotiation.cc:299] Negotiation successful
0925 10:39:10.378592 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 10:39:10.384840 25146 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.1:41861
I20260925 10:39:10.384948 25281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.1:41861 every 8 connection(s)
I20260925 10:39:10.386001 25146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
I20260925 10:39:10.387934 25282 heartbeater.cc:347] Connected to a master server at 127.23.100.62:37929
I20260925 10:39:10.388144 25282 heartbeater.cc:464] Registering TS with master...
I20260925 10:39:10.388731 25282 heartbeater.cc:511] Master 127.23.100.62:37929 requested a full tablet report, sending...
I20260925 10:39:10.390555 25075 ts_manager.cc:194] Registered new tserver with Master: 0311d5f20f3f449db871beefa5ae2330 (127.23.100.1:41861)
I20260925 10:39:10.391891 25075 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.1@KRBTEST.COM'} at 127.23.100.1:58979
I20260925 10:39:10.395552 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 25146
I20260925 10:39:10.395710 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/ts-0/wal/instance
I20260925 10:39:10.396243 23952 external_mini_cluster.cc:1428] Setting key 4dc7abea32f4101c39aa00c14ee60275
WARNING: no policy specified for kudu/127.23.100.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab.
Entry for principal kudu/127.23.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab.
I20260925 10:39:10.431511 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.2:0
--local_ip_for_outbound_sockets=127.23.100.2
--webserver_interface=127.23.100.2
--webserver_port=0
--tserver_master_addrs=127.23.100.62:37929
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab
--principal=kudu/127.23.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:44945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:39:10.736181 25286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:39:10.736474 25286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:39:10.736644 25286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:39:10.743908 25286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:39:10.744046 25286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:39:10.744186 25286 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 10:39:10.744314 25286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.2
I20260925 10:39:10.753471 25286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:44945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab
--principal=kudu/127.23.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.100.62:37929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.2
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:39:10.754755 25286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:39:10.755985 25286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:39:10.768253 25291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:10.768461 25292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:10.768474 25294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:10.769289 25286 server_base.cc:1034] running on GCE node
I20260925 10:39:10.769804 25286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:39:10.770634 25286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:39:10.771855 25286 hybrid_clock.cc:648] HybridClock initialized: now 1790332750771798 us; error 30 us; skew 500 ppm
I20260925 10:39:10.772122 25286 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:39:10 dist-test-slave-dxdd krb5kdc[25030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332750, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:39:10.778440 25286 init.cc:377] Logged in from keytab as kudu/127.23.100.2@KRBTEST.COM (short username kudu)
I20260925 10:39:10.779937 25286 webserver.cc:460] Webserver started at http://127.23.100.2:45267/ using document root <none> and password file <none>
I20260925 10:39:10.780596 25286 fs_manager.cc:359] Metadata directory not provided
I20260925 10:39:10.780712 25286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:39:10.781006 25286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:39:10.783679 25286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/ts-1/data/instance:
uuid: "437b05806f564173921a9823c42e4239"
format_stamp: "Formatted at 2026-09-25 10:39:10 on dist-test-slave-dxdd"
server_key: "3b8bb9bd7c0a0dcc27af9e297a7da7c6"
server_key_iv: "8fa82add9dc4f7af655296fcd7e33043"
server_key_version: "encryptionkey@0"
I20260925 10:39:10.784427 25286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/ts-1/wal/instance:
uuid: "437b05806f564173921a9823c42e4239"
format_stamp: "Formatted at 2026-09-25 10:39:10 on dist-test-slave-dxdd"
server_key: "3b8bb9bd7c0a0dcc27af9e297a7da7c6"
server_key_iv: "8fa82add9dc4f7af655296fcd7e33043"
server_key_version: "encryptionkey@0"
I20260925 10:39:10.789343 25286 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 10:39:10.792573 25301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:10.793761 25286 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:39:10.793927 25286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/ts-1/wal
uuid: "437b05806f564173921a9823c42e4239"
format_stamp: "Formatted at 2026-09-25 10:39:10 on dist-test-slave-dxdd"
server_key: "3b8bb9bd7c0a0dcc27af9e297a7da7c6"
server_key_iv: "8fa82add9dc4f7af655296fcd7e33043"
server_key_version: "encryptionkey@0"
I20260925 10:39:10.794126 25286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:39:10.813687 25286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:39:10.817351 25286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:39:10.817610 25286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:39:10.820242 25286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:39:10.820361 25286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:10.820542 25286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:39:10.820621 25286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 10:39:10 dist-test-slave-dxdd krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332750, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.2@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:39:10.841459 25170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:10.820655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:46987 (local address 127.23.100.62:37929)
0925 10:39:10.820911 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:10.820920 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:10.822270 (+ 1350us) server_negotiation.cc:407] Connection header received
0925 10:39:10.824231 (+ 1961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:10.824239 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:10.824360 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:10.824477 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:10.826832 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:10.827864 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:10.829413 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:10.829818 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:10.829967 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:10.834490 (+ 4523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:10.834495 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:10.834498 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:10.834530 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:10.837670 (+ 3140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:10.838371 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:10.838378 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:10.838382 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:10.838504 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:10.839070 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:10.839073 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:10.839075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:10.839454 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:10.839611 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:10.840642 (+ 1031us) server_negotiation.cc:299] Negotiation successful
0925 10:39:10.840919 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 10:39:10.842208 25310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:10.821246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:37929 (local address 127.23.100.2:46987)
0925 10:39:10.822090 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:10.822136 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:10.823966 (+ 1830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:10.824737 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:10.824748 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:10.825665 (+ 917us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:10.826664 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:10.826687 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:10.828010 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:10.828015 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:10.829268 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:10.829280 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:10.830068 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:10.830076 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:10.830296 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:10.831036 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:10.831081 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:10.834250 (+ 3169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:10.837807 (+ 3557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:10.837816 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:10.837831 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:10.838232 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:10.838612 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:10.838617 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:10.838621 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:10.838917 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:10.839702 (+ 785us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:10.839707 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:10.840216 (+ 509us) client_negotiation.cc:769] Sending connection context
0925 10:39:10.840469 (+ 253us) client_negotiation.cc:240] Negotiation successful
0925 10:39:10.841014 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":372,"threads_started":1}
I20260925 10:39:10.861805 25170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:10.848029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:33873 (local address 127.23.100.62:37929)
0925 10:39:10.848285 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:10.848294 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:10.848790 (+ 496us) server_negotiation.cc:407] Connection header received
0925 10:39:10.848925 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:10.848933 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:10.849028 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:10.849156 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:10.849951 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:10.851340 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:10.853242 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:10.853732 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:10.853855 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:10.856378 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:10.856385 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:10.856391 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:10.856427 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:10.859244 (+ 2817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:10.859929 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:10.859934 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:10.859937 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:10.860052 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:10.860376 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:10.860382 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:10.860385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:10.860689 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:10.860833 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:10.861254 (+ 421us) server_negotiation.cc:299] Negotiation successful
0925 10:39:10.861483 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 10:39:10.862083 25367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:10.848034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:37929 (local address 127.23.100.2:33873)
0925 10:39:10.848701 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:10.848716 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:10.848819 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:10.849163 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:10.849166 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:10.849556 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:10.849824 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:10.849832 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:10.851493 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:10.851503 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:10.853105 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:10.853117 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:10.854054 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:10.854058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:10.854339 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:10.855013 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:10.855043 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:10.856146 (+ 1103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:10.859364 (+ 3218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:10.859370 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:10.859374 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:10.859816 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:10.860153 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:10.860158 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:10.860162 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:10.860278 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:10.860800 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:10.860805 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:10.860978 (+ 173us) client_negotiation.cc:769] Sending connection context
0925 10:39:10.861111 (+ 133us) client_negotiation.cc:240] Negotiation successful
0925 10:39:10.861556 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":379,"threads_started":1}
I20260925 10:39:10.871994 25286 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.2:43899
I20260925 10:39:10.872027 25420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.2:43899 every 8 connection(s)
I20260925 10:39:10.873203 25286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
I20260925 10:39:10.875105 25421 heartbeater.cc:347] Connected to a master server at 127.23.100.62:37929
I20260925 10:39:10.875288 25421 heartbeater.cc:464] Registering TS with master...
I20260925 10:39:10.875840 25421 heartbeater.cc:511] Master 127.23.100.62:37929 requested a full tablet report, sending...
I20260925 10:39:10.876091 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 25286
I20260925 10:39:10.876295 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/ts-1/wal/instance
I20260925 10:39:10.876740 23952 external_mini_cluster.cc:1428] Setting key 11a19397562027e60d85b40350578dec
I20260925 10:39:10.877209 25075 ts_manager.cc:194] Registered new tserver with Master: 437b05806f564173921a9823c42e4239 (127.23.100.2:43899)
I20260925 10:39:10.878216 25075 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.2@KRBTEST.COM'} at 127.23.100.2:46987
WARNING: no policy specified for kudu/127.23.100.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab.
Entry for principal kudu/127.23.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab.
I20260925 10:39:10.915197 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.3:0
--local_ip_for_outbound_sockets=127.23.100.3
--webserver_interface=127.23.100.3
--webserver_port=0
--tserver_master_addrs=127.23.100.62:37929
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab
--principal=kudu/127.23.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:44945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:39:11.232769 25425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:39:11.233093 25425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:39:11.233261 25425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:39:11.240520 25425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:39:11.240648 25425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:39:11.240784 25425 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 10:39:11.240870 25425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.3
I20260925 10:39:11.250737 25425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:44945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab
--principal=kudu/127.23.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.100.62:37929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.3
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:39:11.251892 25425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:39:11.253105 25425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:39:11.266088 25430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:11.266111 25431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:11.266333 25433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:11.266649 25425 server_base.cc:1034] running on GCE node
I20260925 10:39:11.267346 25425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:39:11.268179 25425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:39:11.269418 25425 hybrid_clock.cc:648] HybridClock initialized: now 1790332751269349 us; error 53 us; skew 500 ppm
I20260925 10:39:11.269697 25425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:39:11 dist-test-slave-dxdd krb5kdc[25030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332751, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:39:11.272681 25425 init.cc:377] Logged in from keytab as kudu/127.23.100.3@KRBTEST.COM (short username kudu)
I20260925 10:39:11.274053 25425 webserver.cc:460] Webserver started at http://127.23.100.3:39411/ using document root <none> and password file <none>
I20260925 10:39:11.274746 25425 fs_manager.cc:359] Metadata directory not provided
I20260925 10:39:11.274870 25425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:39:11.275221 25425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:39:11.277920 25425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/ts-2/data/instance:
uuid: "499d386d8b3348f7924f6cf002a08253"
format_stamp: "Formatted at 2026-09-25 10:39:11 on dist-test-slave-dxdd"
server_key: "8c809d62e15d4984f5c15f5697642e04"
server_key_iv: "9bebe3b56e3ff4c1599f470bbc304c36"
server_key_version: "encryptionkey@0"
I20260925 10:39:11.278712 25425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/ts-2/wal/instance:
uuid: "499d386d8b3348f7924f6cf002a08253"
format_stamp: "Formatted at 2026-09-25 10:39:11 on dist-test-slave-dxdd"
server_key: "8c809d62e15d4984f5c15f5697642e04"
server_key_iv: "9bebe3b56e3ff4c1599f470bbc304c36"
server_key_version: "encryptionkey@0"
I20260925 10:39:11.283628 25425 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 10:39:11.286854 25440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:11.288069 25425 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 10:39:11.288232 25425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/ts-2/wal
uuid: "499d386d8b3348f7924f6cf002a08253"
format_stamp: "Formatted at 2026-09-25 10:39:11 on dist-test-slave-dxdd"
server_key: "8c809d62e15d4984f5c15f5697642e04"
server_key_iv: "9bebe3b56e3ff4c1599f470bbc304c36"
server_key_version: "encryptionkey@0"
I20260925 10:39:11.288458 25425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:39:11.305362 25425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:39:11.309136 25425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:39:11.309392 25425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:39:11.312070 25425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:39:11.312214 25425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:11.312371 25425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:39:11.312470 25425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 10:39:11 dist-test-slave-dxdd krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332751, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.3@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:39:11.338948 25170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:11.312672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:38697 (local address 127.23.100.62:37929)
0925 10:39:11.312901 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:11.312907 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:11.314350 (+ 1443us) server_negotiation.cc:407] Connection header received
0925 10:39:11.316515 (+ 2165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:11.316520 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:11.316607 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:11.316735 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:11.319473 (+ 2738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:11.320627 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:11.323996 (+ 3369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:11.324574 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:11.324720 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:11.330262 (+ 5542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:11.330268 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:11.330273 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:11.330364 (+ 91us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:11.334202 (+ 3838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:11.335489 (+ 1287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:11.335497 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:11.335501 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:11.335677 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:11.336337 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:11.336345 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:11.336349 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:11.336816 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:11.336993 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:11.338512 (+ 1519us) server_negotiation.cc:299] Negotiation successful
0925 10:39:11.338686 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 10:39:11.340752 25449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:11.313254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:37929 (local address 127.23.100.3:38697)
0925 10:39:11.314139 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:11.314186 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:11.316187 (+ 2001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:11.317203 (+ 1016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:11.317220 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:11.318230 (+ 1010us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:11.319264 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:11.319295 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:11.320848 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:11.320855 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:11.323800 (+ 2945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:11.323817 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:11.324996 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:11.325004 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:11.325419 (+ 415us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:11.326530 (+ 1111us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:11.326565 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:11.329992 (+ 3427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:11.334427 (+ 4435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:11.334438 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:11.334475 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:11.335274 (+ 799us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:11.335817 (+ 543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:11.335824 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:11.335830 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:11.336178 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:11.336991 (+ 813us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:11.337007 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:11.337631 (+ 624us) client_negotiation.cc:769] Sending connection context
0925 10:39:11.338023 (+ 392us) client_negotiation.cc:240] Negotiation successful
0925 10:39:11.338777 (+ 754us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":390,"threads_started":1}
I20260925 10:39:11.352535 25425 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.3:39069
I20260925 10:39:11.352584 25559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.3:39069 every 8 connection(s)
I20260925 10:39:11.353726 25425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
I20260925 10:39:11.355742 25560 heartbeater.cc:347] Connected to a master server at 127.23.100.62:37929
I20260925 10:39:11.356014 25560 heartbeater.cc:464] Registering TS with master...
I20260925 10:39:11.356828 25560 heartbeater.cc:511] Master 127.23.100.62:37929 requested a full tablet report, sending...
I20260925 10:39:11.358350 25075 ts_manager.cc:194] Registered new tserver with Master: 499d386d8b3348f7924f6cf002a08253 (127.23.100.3:39069)
I20260925 10:39:11.359200 25075 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.3@KRBTEST.COM'} at 127.23.100.3:38697
I20260925 10:39:11.360900 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 25425
I20260925 10:39:11.361083 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/ts-2/wal/instance
I20260925 10:39:11.361497 23952 external_mini_cluster.cc:1428] Setting key a6aab748cb7763aedfeb757cbd4e042e
I20260925 10:39:11.364015 25552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:11.349158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:37929 (local address 127.23.100.3:40847)
0925 10:39:11.350149 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:11.350175 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:11.350427 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:11.350912 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:11.350915 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:11.351547 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:11.352028 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:11.352047 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:11.353236 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:11.353242 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:11.354618 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:11.354631 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:11.355887 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:11.355894 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:11.356101 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:11.356698 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:11.356723 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:11.357743 (+ 1020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:11.361415 (+ 3672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:11.361420 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:11.361424 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:11.361931 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:11.362339 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:11.362344 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:11.362347 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:11.362473 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:11.362954 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:11.362958 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:11.363136 (+ 178us) client_negotiation.cc:769] Sending connection context
0925 10:39:11.363283 (+ 147us) client_negotiation.cc:240] Negotiation successful
0925 10:39:11.363561 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":778,"thread_start_us":581,"threads_started":1}
I20260925 10:39:11.364032 25170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:11.349040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:40847 (local address 127.23.100.62:37929)
0925 10:39:11.349233 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:11.349240 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:11.350268 (+ 1028us) server_negotiation.cc:407] Connection header received
0925 10:39:11.350606 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:11.350613 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:11.350697 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:11.350765 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:11.352261 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:11.353091 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:11.355046 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:11.355700 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:11.355820 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:11.357912 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:11.357918 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:11.357923 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:11.357960 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:11.361264 (+ 3304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:11.362046 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:11.362053 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:11.362056 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:11.362214 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:11.362581 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:11.362587 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:11.362590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:11.362839 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:11.362989 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:11.363405 (+ 416us) server_negotiation.cc:299] Negotiation successful
0925 10:39:11.363617 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 10:39:11.364492 23952 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 10:39:11 dist-test-slave-dxdd krb5kdc[25030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332751, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260925 10:39:11.394276 25282 heartbeater.cc:503] Master 127.23.100.62:37929 was elected leader, sending a full tablet report...
Sep 25 10:39:11 dist-test-slave-dxdd krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332751, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:39:11.407788 25170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:11.395191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56514 (local address 127.23.100.62:37929)
0925 10:39:11.395357 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:11.395361 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:11.395650 (+ 289us) server_negotiation.cc:407] Connection header received
0925 10:39:11.395793 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:11.395799 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:11.395872 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:11.395994 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:11.397354 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:11.398116 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:11.399406 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:11.399788 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:11.399880 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:11.403450 (+ 3570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:11.403455 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:11.403458 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:11.403488 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:11.405724 (+ 2236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:11.406250 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:11.406254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:11.406256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:11.406423 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:11.406717 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:11.406720 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:11.406722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:11.406988 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:11.407165 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:11.407386 (+ 221us) server_negotiation.cc:299] Negotiation successful
0925 10:39:11.407537 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 10:39:11.411584 25075 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56514:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:39:11.418221 25075 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:39:11.443934 25574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:11.429651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53470 (local address 127.23.100.3:39069)
0925 10:39:11.430995 (+ 1344us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:11.431005 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:11.431041 (+ 36us) server_negotiation.cc:407] Connection header received
0925 10:39:11.431147 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:11.431157 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:11.431397 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:11.433597 (+ 2200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:39:11.433984 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:11.437336 (+ 3352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:11.441132 (+ 3796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:11.442795 (+ 1663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:11.442941 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:11.443169 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:11.443404 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 10:39:11.443552 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1142,"thread_start_us":942,"threads_started":1}
I20260925 10:39:11.443929 25573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:11.429407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.3:39069 (local address 127.0.0.1:53470)
0925 10:39:11.430256 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:11.430294 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:11.430524 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:11.431625 (+ 1101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:11.431633 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:11.431681 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:39:11.432151 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:11.432169 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:11.437537 (+ 5368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:11.437545 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:11.440666 (+ 3121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:11.440680 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:11.442966 (+ 2286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:11.442972 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:11.443177 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:11.443231 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 10:39:11.443366 (+ 135us) client_negotiation.cc:240] Negotiation successful
0925 10:39:11.443538 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":448,"threads_started":1}
I20260925 10:39:11.447656 25490 tablet_service.cc:1467] Processing CreateTablet for tablet 13e9727e98324eb2ac1c71e45ff7ebd3 (DEFAULT_TABLE table=test-table [id=ab3d2c88c59c44afa5296c56916cc731]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:39:11.448905 25490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13e9727e98324eb2ac1c71e45ff7ebd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:39:11.449160 25575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:11.430761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:41861 (local address 127.0.0.1:49142)
0925 10:39:11.431262 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:11.431277 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:11.431374 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:11.441743 (+ 10369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:11.441747 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:11.441774 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:39:11.442202 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:11.442216 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:11.444050 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:11.444056 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:11.446449 (+ 2393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:11.446465 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:11.448301 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:11.448307 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:11.448493 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:11.448540 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 10:39:11.448604 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 10:39:11.448734 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"mutex_wait_us":112,"thread_start_us":256,"threads_started":1}
I20260925 10:39:11.449290 25577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:11.431355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.2:43899 (local address 127.0.0.1:42406)
0925 10:39:11.439947 (+ 8592us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:11.439969 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:11.440110 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:11.441549 (+ 1439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:11.441554 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:11.441580 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:39:11.442411 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:11.442420 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:11.444306 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:11.444311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:11.446525 (+ 2214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:11.446539 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:11.448546 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:11.448552 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:11.448714 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:11.448740 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 10:39:11.448810 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 10:39:11.448942 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8415,"mutex_wait_us":19,"thread_start_us":8853,"threads_started":1}
I20260925 10:39:11.449335 25578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:11.431812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42406 (local address 127.23.100.2:43899)
0925 10:39:11.440839 (+ 9027us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:11.440849 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:11.440877 (+ 28us) server_negotiation.cc:407] Connection header received
0925 10:39:11.440971 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:11.440978 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:11.441222 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:11.441431 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:39:11.442571 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:11.444068 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:11.446678 (+ 2610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:11.448071 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:11.448215 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:11.448418 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:11.448912 (+ 494us) server_negotiation.cc:299] Negotiation successful
0925 10:39:11.449075 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8826,"thread_start_us":8698,"threads_started":1}
I20260925 10:39:11.450472 25576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:11.431080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49142 (local address 127.23.100.1:41861)
0925 10:39:11.441152 (+ 10072us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:11.441160 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:11.441186 (+ 26us) server_negotiation.cc:407] Connection header received
0925 10:39:11.441282 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:11.441289 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:11.441521 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:11.442092 (+ 571us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:39:11.442383 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:11.443885 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:11.446647 (+ 2762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:11.448071 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:11.449757 (+ 1686us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:11.449939 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:11.450005 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 10:39:11.450159 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9899,"thread_start_us":9725,"threads_started":1}
I20260925 10:39:11.452732 25351 tablet_service.cc:1467] Processing CreateTablet for tablet 13e9727e98324eb2ac1c71e45ff7ebd3 (DEFAULT_TABLE table=test-table [id=ab3d2c88c59c44afa5296c56916cc731]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:39:11.454067 25351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13e9727e98324eb2ac1c71e45ff7ebd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:39:11.454507 25212 tablet_service.cc:1467] Processing CreateTablet for tablet 13e9727e98324eb2ac1c71e45ff7ebd3 (DEFAULT_TABLE table=test-table [id=ab3d2c88c59c44afa5296c56916cc731]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:39:11.455561 25212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13e9727e98324eb2ac1c71e45ff7ebd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:39:11.463804 25579 tablet_bootstrap.cc:492] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 499d386d8b3348f7924f6cf002a08253: Bootstrap starting.
I20260925 10:39:11.467628 25580 tablet_bootstrap.cc:492] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 437b05806f564173921a9823c42e4239: Bootstrap starting.
I20260925 10:39:11.468767 25579 tablet_bootstrap.cc:654] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 499d386d8b3348f7924f6cf002a08253: Neither blocks nor log segments found. Creating new log.
I20260925 10:39:11.469040 25581 tablet_bootstrap.cc:492] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 0311d5f20f3f449db871beefa5ae2330: Bootstrap starting.
I20260925 10:39:11.469978 25579 log.cc:824] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 499d386d8b3348f7924f6cf002a08253: Log is configured to *not* fsync() on all Append() calls
I20260925 10:39:11.471181 25580 tablet_bootstrap.cc:654] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 437b05806f564173921a9823c42e4239: Neither blocks nor log segments found. Creating new log.
I20260925 10:39:11.472467 25580 log.cc:824] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 437b05806f564173921a9823c42e4239: Log is configured to *not* fsync() on all Append() calls
I20260925 10:39:11.472522 25579 tablet_bootstrap.cc:492] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 499d386d8b3348f7924f6cf002a08253: No bootstrap required, opened a new log
I20260925 10:39:11.472751 25581 tablet_bootstrap.cc:654] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 0311d5f20f3f449db871beefa5ae2330: Neither blocks nor log segments found. Creating new log.
I20260925 10:39:11.472817 25579 ts_tablet_manager.cc:1397] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 499d386d8b3348f7924f6cf002a08253: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260925 10:39:11.473865 25581 log.cc:824] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 0311d5f20f3f449db871beefa5ae2330: Log is configured to *not* fsync() on all Append() calls
I20260925 10:39:11.475299 25580 tablet_bootstrap.cc:492] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 437b05806f564173921a9823c42e4239: No bootstrap required, opened a new log
I20260925 10:39:11.475565 25580 ts_tablet_manager.cc:1397] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 437b05806f564173921a9823c42e4239: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 10:39:11.476696 25581 tablet_bootstrap.cc:492] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 0311d5f20f3f449db871beefa5ae2330: No bootstrap required, opened a new log
I20260925 10:39:11.476944 25581 ts_tablet_manager.cc:1397] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 0311d5f20f3f449db871beefa5ae2330: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260925 10:39:11.480786 25579 raft_consensus.cc:348] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 499d386d8b3348f7924f6cf002a08253 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "499d386d8b3348f7924f6cf002a08253" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 39069 } } peers { permanent_uuid: "0311d5f20f3f449db871beefa5ae2330" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 41861 } } peers { permanent_uuid: "437b05806f564173921a9823c42e4239" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 43899 } }
I20260925 10:39:11.481124 25579 raft_consensus.cc:374] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 499d386d8b3348f7924f6cf002a08253 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:39:11.481247 25579 raft_consensus.cc:729] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 499d386d8b3348f7924f6cf002a08253 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 499d386d8b3348f7924f6cf002a08253, State: Initialized, Role: FOLLOWER
I20260925 10:39:11.481818 25579 consensus_queue.cc:260] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 499d386d8b3348f7924f6cf002a08253 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "499d386d8b3348f7924f6cf002a08253" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 39069 } } peers { permanent_uuid: "0311d5f20f3f449db871beefa5ae2330" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 41861 } } peers { permanent_uuid: "437b05806f564173921a9823c42e4239" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 43899 } }
I20260925 10:39:11.482865 25560 heartbeater.cc:503] Master 127.23.100.62:37929 was elected leader, sending a full tablet report...
I20260925 10:39:11.484581 25579 ts_tablet_manager.cc:1428] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 499d386d8b3348f7924f6cf002a08253: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260925 10:39:11.485965 25580 raft_consensus.cc:348] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 437b05806f564173921a9823c42e4239 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "499d386d8b3348f7924f6cf002a08253" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 39069 } } peers { permanent_uuid: "0311d5f20f3f449db871beefa5ae2330" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 41861 } } peers { permanent_uuid: "437b05806f564173921a9823c42e4239" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 43899 } }
I20260925 10:39:11.486385 25580 raft_consensus.cc:374] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 437b05806f564173921a9823c42e4239 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:39:11.486483 25580 raft_consensus.cc:729] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 437b05806f564173921a9823c42e4239 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 437b05806f564173921a9823c42e4239, State: Initialized, Role: FOLLOWER
I20260925 10:39:11.487015 25580 consensus_queue.cc:260] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 437b05806f564173921a9823c42e4239 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "499d386d8b3348f7924f6cf002a08253" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 39069 } } peers { permanent_uuid: "0311d5f20f3f449db871beefa5ae2330" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 41861 } } peers { permanent_uuid: "437b05806f564173921a9823c42e4239" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 43899 } }
I20260925 10:39:11.487525 25581 raft_consensus.cc:348] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 0311d5f20f3f449db871beefa5ae2330 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "499d386d8b3348f7924f6cf002a08253" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 39069 } } peers { permanent_uuid: "0311d5f20f3f449db871beefa5ae2330" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 41861 } } peers { permanent_uuid: "437b05806f564173921a9823c42e4239" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 43899 } }
I20260925 10:39:11.487993 25581 raft_consensus.cc:374] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 0311d5f20f3f449db871beefa5ae2330 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:39:11.488039 25421 heartbeater.cc:503] Master 127.23.100.62:37929 was elected leader, sending a full tablet report...
I20260925 10:39:11.488112 25581 raft_consensus.cc:729] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 0311d5f20f3f449db871beefa5ae2330 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0311d5f20f3f449db871beefa5ae2330, State: Initialized, Role: FOLLOWER
I20260925 10:39:11.488647 25581 consensus_queue.cc:260] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 0311d5f20f3f449db871beefa5ae2330 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "499d386d8b3348f7924f6cf002a08253" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 39069 } } peers { permanent_uuid: "0311d5f20f3f449db871beefa5ae2330" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 41861 } } peers { permanent_uuid: "437b05806f564173921a9823c42e4239" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 43899 } }
I20260925 10:39:11.489825 25580 ts_tablet_manager.cc:1428] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 437b05806f564173921a9823c42e4239: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260925 10:39:11.490255 25581 ts_tablet_manager.cc:1428] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 0311d5f20f3f449db871beefa5ae2330: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
W20260925 10:39:11.605605 25561 tablet.cc:2369] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 499d386d8b3348f7924f6cf002a08253: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:39:11.625178 25422 tablet.cc:2369] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 437b05806f564173921a9823c42e4239: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:39:11.638489 25283 tablet.cc:2369] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 0311d5f20f3f449db871beefa5ae2330: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:39:11.728071 25586 raft_consensus.cc:482] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 437b05806f564173921a9823c42e4239 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:39:11.728494 25586 raft_consensus.cc:504] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 437b05806f564173921a9823c42e4239 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "499d386d8b3348f7924f6cf002a08253" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 39069 } } peers { permanent_uuid: "0311d5f20f3f449db871beefa5ae2330" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 41861 } } peers { permanent_uuid: "437b05806f564173921a9823c42e4239" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 43899 } }
I20260925 10:39:11.730165 25586 leader_election.cc:288] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 437b05806f564173921a9823c42e4239 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 499d386d8b3348f7924f6cf002a08253 (127.23.100.3:39069), 0311d5f20f3f449db871beefa5ae2330 (127.23.100.1:41861)
I20260925 10:39:11.736613 25574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:11.730510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:45833 (local address 127.23.100.3:39069)
0925 10:39:11.730706 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:11.730710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:11.731170 (+ 460us) server_negotiation.cc:407] Connection header received
0925 10:39:11.731517 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:11.731523 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:11.731606 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:11.731669 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:39:11.732255 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:11.733276 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:11.734988 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:11.735723 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:11.735910 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:11.735997 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:11.736237 (+ 240us) server_negotiation.cc:299] Negotiation successful
0925 10:39:11.736359 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 10:39:11.736675 25588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:11.730403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.3:39069 (local address 127.23.100.2:45833)
0925 10:39:11.731072 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:11.731089 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:11.731300 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:11.731826 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:11.731829 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:11.731848 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:39:11.732120 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:11.732127 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:11.733390 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:11.733393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:11.734837 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:11.734847 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:11.735924 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:11.735930 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:11.736088 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:11.736111 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 10:39:11.736212 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 10:39:11.736339 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":314,"threads_started":1}
I20260925 10:39:11.736675 25587 raft_consensus.cc:482] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 0311d5f20f3f449db871beefa5ae2330 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:39:11.737092 25576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:11.731357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:55391 (local address 127.23.100.1:41861)
0925 10:39:11.731589 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:11.731595 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:11.731820 (+ 225us) server_negotiation.cc:407] Connection header received
0925 10:39:11.732089 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:11.732095 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:11.732157 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:11.732264 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:39:11.732693 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:11.733590 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:11.735319 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:11.736074 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:11.736274 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:11.736342 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:11.736680 (+ 338us) server_negotiation.cc:299] Negotiation successful
0925 10:39:11.736818 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 10:39:11.737524 25589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:11.731134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:41861 (local address 127.23.100.2:55391)
0925 10:39:11.731718 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:11.731732 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:11.731944 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:11.732264 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:11.732267 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:11.732284 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:39:11.732551 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:11.732557 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:11.733719 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:11.733724 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:11.735185 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:11.735196 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:11.736289 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:11.736295 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:11.736475 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:11.736512 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 10:39:11.737003 (+ 491us) client_negotiation.cc:240] Negotiation successful
0925 10:39:11.737137 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"spinlock_wait_cycles":21888,"thread_start_us":287,"threads_started":1}
I20260925 10:39:11.736960 25587 raft_consensus.cc:504] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 0311d5f20f3f449db871beefa5ae2330 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "499d386d8b3348f7924f6cf002a08253" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 39069 } } peers { permanent_uuid: "0311d5f20f3f449db871beefa5ae2330" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 41861 } } peers { permanent_uuid: "437b05806f564173921a9823c42e4239" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 43899 } }
I20260925 10:39:11.737646 25510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13e9727e98324eb2ac1c71e45ff7ebd3" candidate_uuid: "437b05806f564173921a9823c42e4239" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "499d386d8b3348f7924f6cf002a08253" is_pre_election: true
I20260925 10:39:11.738152 25510 raft_consensus.cc:2454] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 499d386d8b3348f7924f6cf002a08253 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 437b05806f564173921a9823c42e4239 in term 0.
I20260925 10:39:11.738392 25236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13e9727e98324eb2ac1c71e45ff7ebd3" candidate_uuid: "437b05806f564173921a9823c42e4239" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0311d5f20f3f449db871beefa5ae2330" is_pre_election: true
I20260925 10:39:11.738791 25302 leader_election.cc:302] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 437b05806f564173921a9823c42e4239 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 437b05806f564173921a9823c42e4239, 499d386d8b3348f7924f6cf002a08253; no voters:
I20260925 10:39:11.738871 25236 raft_consensus.cc:2454] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 0311d5f20f3f449db871beefa5ae2330 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 437b05806f564173921a9823c42e4239 in term 0.
I20260925 10:39:11.739147 25586 raft_consensus.cc:2781] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 437b05806f564173921a9823c42e4239 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:39:11.739279 25586 raft_consensus.cc:482] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 437b05806f564173921a9823c42e4239 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:39:11.739396 25586 raft_consensus.cc:3037] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 437b05806f564173921a9823c42e4239 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:39:11.739665 25587 leader_election.cc:288] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 0311d5f20f3f449db871beefa5ae2330 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 499d386d8b3348f7924f6cf002a08253 (127.23.100.3:39069), 437b05806f564173921a9823c42e4239 (127.23.100.2:43899)
I20260925 10:39:11.742817 25586 raft_consensus.cc:504] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 437b05806f564173921a9823c42e4239 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "499d386d8b3348f7924f6cf002a08253" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 39069 } } peers { permanent_uuid: "0311d5f20f3f449db871beefa5ae2330" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 41861 } } peers { permanent_uuid: "437b05806f564173921a9823c42e4239" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 43899 } }
I20260925 10:39:11.743558 25586 leader_election.cc:288] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 437b05806f564173921a9823c42e4239 [CANDIDATE]: Term 1 election: Requested vote from peers 499d386d8b3348f7924f6cf002a08253 (127.23.100.3:39069), 0311d5f20f3f449db871beefa5ae2330 (127.23.100.1:41861)
I20260925 10:39:11.743966 25236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13e9727e98324eb2ac1c71e45ff7ebd3" candidate_uuid: "437b05806f564173921a9823c42e4239" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0311d5f20f3f449db871beefa5ae2330"
I20260925 10:39:11.744081 25510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13e9727e98324eb2ac1c71e45ff7ebd3" candidate_uuid: "437b05806f564173921a9823c42e4239" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "499d386d8b3348f7924f6cf002a08253"
I20260925 10:39:11.744204 25236 raft_consensus.cc:3037] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 0311d5f20f3f449db871beefa5ae2330 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:39:11.744257 25510 raft_consensus.cc:3037] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 499d386d8b3348f7924f6cf002a08253 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:39:11.747529 25510 raft_consensus.cc:2454] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 499d386d8b3348f7924f6cf002a08253 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 437b05806f564173921a9823c42e4239 in term 1.
I20260925 10:39:11.747866 25302 leader_election.cc:302] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 437b05806f564173921a9823c42e4239 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 437b05806f564173921a9823c42e4239, 499d386d8b3348f7924f6cf002a08253; no voters:
I20260925 10:39:11.748013 25590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:11.739812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.3:39069 (local address 127.23.100.1:47961)
0925 10:39:11.740333 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:11.740351 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:11.740468 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:11.741026 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:11.741031 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:11.741059 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:39:11.741339 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:11.741346 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:11.742774 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:11.742780 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:11.745278 (+ 2498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:11.745291 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:11.747135 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:11.747141 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:11.747324 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:11.747367 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 10:39:11.747465 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 10:39:11.747609 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"mutex_wait_us":48,"thread_start_us":446,"threads_started":1}
I20260925 10:39:11.748179 25236 raft_consensus.cc:2454] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 0311d5f20f3f449db871beefa5ae2330 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 437b05806f564173921a9823c42e4239 in term 1.
I20260925 10:39:11.748521 25574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:11.740210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:47961 (local address 127.23.100.3:39069)
0925 10:39:11.740374 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:11.740381 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:11.740410 (+ 29us) server_negotiation.cc:407] Connection header received
0925 10:39:11.740710 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:11.740717 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:11.740792 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:11.740875 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:39:11.741466 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:11.742628 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:11.745464 (+ 2836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:11.746791 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:11.746931 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:11.746995 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:11.748055 (+ 1060us) server_negotiation.cc:299] Negotiation successful
0925 10:39:11.748282 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 10:39:11.748876 25586 raft_consensus.cc:2781] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 437b05806f564173921a9823c42e4239 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:39:11.749022 25510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13e9727e98324eb2ac1c71e45ff7ebd3" candidate_uuid: "0311d5f20f3f449db871beefa5ae2330" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "499d386d8b3348f7924f6cf002a08253" is_pre_election: true
I20260925 10:39:11.749142 25578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:11.740879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:38853 (local address 127.23.100.2:43899)
0925 10:39:11.741066 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:11.741071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:11.741965 (+ 894us) server_negotiation.cc:407] Connection header received
0925 10:39:11.742160 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:11.742165 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:11.742226 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:11.742349 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:39:11.743016 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:11.744046 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:11.746714 (+ 2668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:11.747960 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:11.748090 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:11.748174 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:11.748735 (+ 561us) server_negotiation.cc:299] Negotiation successful
0925 10:39:11.748868 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 10:39:11.749297 25510 raft_consensus.cc:2379] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 499d386d8b3348f7924f6cf002a08253 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0311d5f20f3f449db871beefa5ae2330 in current term 1: Already voted for candidate 437b05806f564173921a9823c42e4239 in this term.
I20260925 10:39:11.750123 25591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:11.740626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.2:43899 (local address 127.23.100.1:38853)
0925 10:39:11.741878 (+ 1252us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:11.741896 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:11.742051 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:11.742380 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:11.742384 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:11.742415 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:39:11.742810 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:11.742821 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:11.744228 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:11.744233 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:11.746537 (+ 2304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:11.746557 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:11.748375 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:11.748382 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:11.748572 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:11.748601 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 10:39:11.749380 (+ 779us) client_negotiation.cc:240] Negotiation successful
0925 10:39:11.749568 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1115,"mutex_wait_us":29,"thread_start_us":955,"threads_started":1}
I20260925 10:39:11.751138 25376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13e9727e98324eb2ac1c71e45ff7ebd3" candidate_uuid: "0311d5f20f3f449db871beefa5ae2330" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "437b05806f564173921a9823c42e4239" is_pre_election: true
I20260925 10:39:11.754552 25586 raft_consensus.cc:686] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 437b05806f564173921a9823c42e4239 [term 1 LEADER]: Becoming Leader. State: Replica: 437b05806f564173921a9823c42e4239, State: Running, Role: LEADER
I20260925 10:39:11.755002 25586 consensus_queue.cc:237] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 437b05806f564173921a9823c42e4239 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "499d386d8b3348f7924f6cf002a08253" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 39069 } } peers { permanent_uuid: "0311d5f20f3f449db871beefa5ae2330" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 41861 } } peers { permanent_uuid: "437b05806f564173921a9823c42e4239" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 43899 } }
I20260925 10:39:11.759121 25162 leader_election.cc:302] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 0311d5f20f3f449db871beefa5ae2330 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0311d5f20f3f449db871beefa5ae2330; no voters: 437b05806f564173921a9823c42e4239, 499d386d8b3348f7924f6cf002a08253
I20260925 10:39:11.759536 25587 raft_consensus.cc:2726] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 0311d5f20f3f449db871beefa5ae2330 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 10:39:11.760244 25073 catalog_manager.cc:5495] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 437b05806f564173921a9823c42e4239 reported cstate change: term changed from 0 to 1, leader changed from <none> to 437b05806f564173921a9823c42e4239 (127.23.100.2). New cstate: current_term: 1 leader_uuid: "437b05806f564173921a9823c42e4239" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "499d386d8b3348f7924f6cf002a08253" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 39069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0311d5f20f3f449db871beefa5ae2330" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 41861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "437b05806f564173921a9823c42e4239" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 43899 } health_report { overall_health: HEALTHY } } }
I20260925 10:39:11.808378 25578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:11.803039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42420 (local address 127.23.100.2:43899)
0925 10:39:11.803230 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:11.803237 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:11.803283 (+ 46us) server_negotiation.cc:407] Connection header received
0925 10:39:11.803621 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:11.803625 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:11.803698 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:11.803832 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:39:11.804212 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:11.804958 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:11.806264 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:11.806711 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:11.806834 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:11.807302 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:39:11.807781 (+ 479us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:39:11.807933 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:11.808036 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 10:39:11.808116 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 10:39:11.815734 25510 raft_consensus.cc:1261] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 499d386d8b3348f7924f6cf002a08253 [term 1 FOLLOWER]: Refusing update from remote peer 437b05806f564173921a9823c42e4239: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:39:11.815773 25236 raft_consensus.cc:1261] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 0311d5f20f3f449db871beefa5ae2330 [term 1 FOLLOWER]: Refusing update from remote peer 437b05806f564173921a9823c42e4239: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:39:11.816480 25592 consensus_queue.cc:1035] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 437b05806f564173921a9823c42e4239 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0311d5f20f3f449db871beefa5ae2330" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 41861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:39:11.816802 25586 consensus_queue.cc:1035] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 437b05806f564173921a9823c42e4239 [LEADER]: Connected to new peer: Peer: permanent_uuid: "499d386d8b3348f7924f6cf002a08253" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 39069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:39:11.827939 25595 mvcc.cc:205] Tried to move back new op lower bound from 7333202951428870144 to 7333202951192461312. Current Snapshot: MvccSnapshot[applied={T|T < 7333202951428870144}]
I20260925 10:39:11.852445 25074 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:39:11.852756 25074 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:39:11.856168 25074 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 17626f350f1e4739a7e884a2be1a8823: Sending DeleteTablet for 3 replicas of tablet 13e9727e98324eb2ac1c71e45ff7ebd3
I20260925 10:39:11.857311 25490 tablet_service.cc:1514] Processing DeleteTablet for tablet 13e9727e98324eb2ac1c71e45ff7ebd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:39:11 UTC) from {username='kudu', principal='kudu/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:53470
I20260925 10:39:11.857426 25351 tablet_service.cc:1514] Processing DeleteTablet for tablet 13e9727e98324eb2ac1c71e45ff7ebd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:39:11 UTC) from {username='kudu', principal='kudu/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:42406
I20260925 10:39:11.857563 25212 tablet_service.cc:1514] Processing DeleteTablet for tablet 13e9727e98324eb2ac1c71e45ff7ebd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:39:11 UTC) from {username='kudu', principal='kudu/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:49142
I20260925 10:39:11.859097 25074 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56514:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:39:11.860083 25074 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:39:11.870822 25607 tablet_replica.cc:330] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 437b05806f564173921a9823c42e4239: stopping tablet replica
I20260925 10:39:11.870744 25350 tablet_service.cc:1467] Processing CreateTablet for tablet f69bc748f3cb4ef79e77073b8e3c406f (DEFAULT_TABLE table=test-table [id=24964006bfb947558428d4b81dc3a617]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:39:11.870860 25211 tablet_service.cc:1467] Processing CreateTablet for tablet f69bc748f3cb4ef79e77073b8e3c406f (DEFAULT_TABLE table=test-table [id=24964006bfb947558428d4b81dc3a617]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:39:11.871316 25350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f69bc748f3cb4ef79e77073b8e3c406f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:39:11.871567 25211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f69bc748f3cb4ef79e77073b8e3c406f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:39:11.875113 25489 tablet_service.cc:1467] Processing CreateTablet for tablet f69bc748f3cb4ef79e77073b8e3c406f (DEFAULT_TABLE table=test-table [id=24964006bfb947558428d4b81dc3a617]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:39:11.875641 25489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f69bc748f3cb4ef79e77073b8e3c406f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:39:11.878114 25580 tablet_bootstrap.cc:492] T f69bc748f3cb4ef79e77073b8e3c406f P 437b05806f564173921a9823c42e4239: Bootstrap starting.
I20260925 10:39:11.878592 25607 raft_consensus.cc:2229] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 437b05806f564173921a9823c42e4239 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:39:11.879333 25607 raft_consensus.cc:2258] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 437b05806f564173921a9823c42e4239 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:39:11.880125 25580 tablet_bootstrap.cc:654] T f69bc748f3cb4ef79e77073b8e3c406f P 437b05806f564173921a9823c42e4239: Neither blocks nor log segments found. Creating new log.
I20260925 10:39:11.880594 25608 tablet_replica.cc:330] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 0311d5f20f3f449db871beefa5ae2330: stopping tablet replica
I20260925 10:39:11.881083 25608 raft_consensus.cc:2229] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 0311d5f20f3f449db871beefa5ae2330 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:39:11.881723 25608 raft_consensus.cc:2258] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 0311d5f20f3f449db871beefa5ae2330 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:39:11.883378 25580 tablet_bootstrap.cc:492] T f69bc748f3cb4ef79e77073b8e3c406f P 437b05806f564173921a9823c42e4239: No bootstrap required, opened a new log
I20260925 10:39:11.883576 25580 ts_tablet_manager.cc:1397] T f69bc748f3cb4ef79e77073b8e3c406f P 437b05806f564173921a9823c42e4239: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260925 10:39:11.884069 25608 ts_tablet_manager.cc:1905] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 0311d5f20f3f449db871beefa5ae2330: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:39:11.885514 25580 raft_consensus.cc:348] T f69bc748f3cb4ef79e77073b8e3c406f P 437b05806f564173921a9823c42e4239 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311d5f20f3f449db871beefa5ae2330" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 41861 } } peers { permanent_uuid: "437b05806f564173921a9823c42e4239" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 43899 } } peers { permanent_uuid: "499d386d8b3348f7924f6cf002a08253" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 39069 } }
I20260925 10:39:11.885815 25580 raft_consensus.cc:374] T f69bc748f3cb4ef79e77073b8e3c406f P 437b05806f564173921a9823c42e4239 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:39:11.885962 25580 raft_consensus.cc:729] T f69bc748f3cb4ef79e77073b8e3c406f P 437b05806f564173921a9823c42e4239 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 437b05806f564173921a9823c42e4239, State: Initialized, Role: FOLLOWER
I20260925 10:39:11.886032 25579 tablet_bootstrap.cc:492] T f69bc748f3cb4ef79e77073b8e3c406f P 499d386d8b3348f7924f6cf002a08253: Bootstrap starting.
I20260925 10:39:11.886338 25580 consensus_queue.cc:260] T f69bc748f3cb4ef79e77073b8e3c406f P 437b05806f564173921a9823c42e4239 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311d5f20f3f449db871beefa5ae2330" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 41861 } } peers { permanent_uuid: "437b05806f564173921a9823c42e4239" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 43899 } } peers { permanent_uuid: "499d386d8b3348f7924f6cf002a08253" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 39069 } }
I20260925 10:39:11.887679 25579 tablet_bootstrap.cc:654] T f69bc748f3cb4ef79e77073b8e3c406f P 499d386d8b3348f7924f6cf002a08253: Neither blocks nor log segments found. Creating new log.
I20260925 10:39:11.887698 25580 ts_tablet_manager.cc:1428] T f69bc748f3cb4ef79e77073b8e3c406f P 437b05806f564173921a9823c42e4239: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260925 10:39:11.898886 25607 ts_tablet_manager.cc:1905] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 437b05806f564173921a9823c42e4239: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:39:11.901525 25581 tablet_bootstrap.cc:492] T f69bc748f3cb4ef79e77073b8e3c406f P 0311d5f20f3f449db871beefa5ae2330: Bootstrap starting.
I20260925 10:39:11.903254 25581 tablet_bootstrap.cc:654] T f69bc748f3cb4ef79e77073b8e3c406f P 0311d5f20f3f449db871beefa5ae2330: Neither blocks nor log segments found. Creating new log.
I20260925 10:39:11.909014 25606 tablet_replica.cc:330] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 499d386d8b3348f7924f6cf002a08253: stopping tablet replica
I20260925 10:39:11.909476 25606 raft_consensus.cc:2229] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 499d386d8b3348f7924f6cf002a08253 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:39:11.909945 25606 raft_consensus.cc:2258] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 499d386d8b3348f7924f6cf002a08253 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:39:11.914162 25606 ts_tablet_manager.cc:1905] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 499d386d8b3348f7924f6cf002a08253: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:39:11.915282 25579 tablet_bootstrap.cc:492] T f69bc748f3cb4ef79e77073b8e3c406f P 499d386d8b3348f7924f6cf002a08253: No bootstrap required, opened a new log
I20260925 10:39:11.915442 25579 ts_tablet_manager.cc:1397] T f69bc748f3cb4ef79e77073b8e3c406f P 499d386d8b3348f7924f6cf002a08253: Time spent bootstrapping tablet: real 0.029s user 0.009s sys 0.006s
I20260925 10:39:11.916241 25579 raft_consensus.cc:348] T f69bc748f3cb4ef79e77073b8e3c406f P 499d386d8b3348f7924f6cf002a08253 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311d5f20f3f449db871beefa5ae2330" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 41861 } } peers { permanent_uuid: "437b05806f564173921a9823c42e4239" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 43899 } } peers { permanent_uuid: "499d386d8b3348f7924f6cf002a08253" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 39069 } }
I20260925 10:39:11.916483 25579 raft_consensus.cc:374] T f69bc748f3cb4ef79e77073b8e3c406f P 499d386d8b3348f7924f6cf002a08253 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:39:11.916594 25579 raft_consensus.cc:729] T f69bc748f3cb4ef79e77073b8e3c406f P 499d386d8b3348f7924f6cf002a08253 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 499d386d8b3348f7924f6cf002a08253, State: Initialized, Role: FOLLOWER
I20260925 10:39:11.916869 25579 consensus_queue.cc:260] T f69bc748f3cb4ef79e77073b8e3c406f P 499d386d8b3348f7924f6cf002a08253 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311d5f20f3f449db871beefa5ae2330" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 41861 } } peers { permanent_uuid: "437b05806f564173921a9823c42e4239" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 43899 } } peers { permanent_uuid: "499d386d8b3348f7924f6cf002a08253" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 39069 } }
I20260925 10:39:11.917526 25579 ts_tablet_manager.cc:1428] T f69bc748f3cb4ef79e77073b8e3c406f P 499d386d8b3348f7924f6cf002a08253: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:39:11.921990 25608 ts_tablet_manager.cc:1918] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 0311d5f20f3f449db871beefa5ae2330: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:39:11.921990 25607 ts_tablet_manager.cc:1918] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 437b05806f564173921a9823c42e4239: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:39:11.922163 25607 log.cc:1196] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 437b05806f564173921a9823c42e4239: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/ts-1/wal/wals/13e9727e98324eb2ac1c71e45ff7ebd3
I20260925 10:39:11.922855 25607 ts_tablet_manager.cc:1939] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 437b05806f564173921a9823c42e4239: Deleting consensus metadata
I20260925 10:39:11.923848 25606 ts_tablet_manager.cc:1918] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 499d386d8b3348f7924f6cf002a08253: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:39:11.924101 25608 log.cc:1196] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 0311d5f20f3f449db871beefa5ae2330: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/ts-0/wal/wals/13e9727e98324eb2ac1c71e45ff7ebd3
I20260925 10:39:11.924750 25608 ts_tablet_manager.cc:1939] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 0311d5f20f3f449db871beefa5ae2330: Deleting consensus metadata
I20260925 10:39:11.924749 25059 catalog_manager.cc:4841] TS 437b05806f564173921a9823c42e4239 (127.23.100.2:43899): tablet 13e9727e98324eb2ac1c71e45ff7ebd3 (table test-table [id=ab3d2c88c59c44afa5296c56916cc731]) successfully deleted
I20260925 10:39:11.925057 25581 tablet_bootstrap.cc:492] T f69bc748f3cb4ef79e77073b8e3c406f P 0311d5f20f3f449db871beefa5ae2330: No bootstrap required, opened a new log
I20260925 10:39:11.925208 25606 log.cc:1196] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 499d386d8b3348f7924f6cf002a08253: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/ts-2/wal/wals/13e9727e98324eb2ac1c71e45ff7ebd3
I20260925 10:39:11.925343 25581 ts_tablet_manager.cc:1397] T f69bc748f3cb4ef79e77073b8e3c406f P 0311d5f20f3f449db871beefa5ae2330: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.005s
I20260925 10:39:11.926283 25606 ts_tablet_manager.cc:1939] T 13e9727e98324eb2ac1c71e45ff7ebd3 P 499d386d8b3348f7924f6cf002a08253: Deleting consensus metadata
I20260925 10:39:11.926748 25059 catalog_manager.cc:4841] TS 0311d5f20f3f449db871beefa5ae2330 (127.23.100.1:41861): tablet 13e9727e98324eb2ac1c71e45ff7ebd3 (table test-table [id=ab3d2c88c59c44afa5296c56916cc731]) successfully deleted
I20260925 10:39:11.927100 25581 raft_consensus.cc:348] T f69bc748f3cb4ef79e77073b8e3c406f P 0311d5f20f3f449db871beefa5ae2330 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311d5f20f3f449db871beefa5ae2330" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 41861 } } peers { permanent_uuid: "437b05806f564173921a9823c42e4239" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 43899 } } peers { permanent_uuid: "499d386d8b3348f7924f6cf002a08253" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 39069 } }
I20260925 10:39:11.927338 25581 raft_consensus.cc:374] T f69bc748f3cb4ef79e77073b8e3c406f P 0311d5f20f3f449db871beefa5ae2330 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:39:11.927454 25581 raft_consensus.cc:729] T f69bc748f3cb4ef79e77073b8e3c406f P 0311d5f20f3f449db871beefa5ae2330 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0311d5f20f3f449db871beefa5ae2330, State: Initialized, Role: FOLLOWER
I20260925 10:39:11.927726 25581 consensus_queue.cc:260] T f69bc748f3cb4ef79e77073b8e3c406f P 0311d5f20f3f449db871beefa5ae2330 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311d5f20f3f449db871beefa5ae2330" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 41861 } } peers { permanent_uuid: "437b05806f564173921a9823c42e4239" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 43899 } } peers { permanent_uuid: "499d386d8b3348f7924f6cf002a08253" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 39069 } }
I20260925 10:39:11.927883 25059 catalog_manager.cc:4841] TS 499d386d8b3348f7924f6cf002a08253 (127.23.100.3:39069): tablet 13e9727e98324eb2ac1c71e45ff7ebd3 (table test-table [id=ab3d2c88c59c44afa5296c56916cc731]) successfully deleted
I20260925 10:39:11.928452 25581 ts_tablet_manager.cc:1428] T f69bc748f3cb4ef79e77073b8e3c406f P 0311d5f20f3f449db871beefa5ae2330: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260925 10:39:11.928689 25604 raft_consensus.cc:482] T f69bc748f3cb4ef79e77073b8e3c406f P 437b05806f564173921a9823c42e4239 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:39:11.928968 25604 raft_consensus.cc:504] T f69bc748f3cb4ef79e77073b8e3c406f P 437b05806f564173921a9823c42e4239 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311d5f20f3f449db871beefa5ae2330" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 41861 } } peers { permanent_uuid: "437b05806f564173921a9823c42e4239" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 43899 } } peers { permanent_uuid: "499d386d8b3348f7924f6cf002a08253" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 39069 } }
I20260925 10:39:11.929675 25604 leader_election.cc:288] T f69bc748f3cb4ef79e77073b8e3c406f P 437b05806f564173921a9823c42e4239 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0311d5f20f3f449db871beefa5ae2330 (127.23.100.1:41861), 499d386d8b3348f7924f6cf002a08253 (127.23.100.3:39069)
I20260925 10:39:11.930167 25510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69bc748f3cb4ef79e77073b8e3c406f" candidate_uuid: "437b05806f564173921a9823c42e4239" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "499d386d8b3348f7924f6cf002a08253" is_pre_election: true
I20260925 10:39:11.930187 25236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69bc748f3cb4ef79e77073b8e3c406f" candidate_uuid: "437b05806f564173921a9823c42e4239" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0311d5f20f3f449db871beefa5ae2330" is_pre_election: true
I20260925 10:39:11.930414 25510 raft_consensus.cc:2454] T f69bc748f3cb4ef79e77073b8e3c406f P 499d386d8b3348f7924f6cf002a08253 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 437b05806f564173921a9823c42e4239 in term 0.
I20260925 10:39:11.930477 25236 raft_consensus.cc:2454] T f69bc748f3cb4ef79e77073b8e3c406f P 0311d5f20f3f449db871beefa5ae2330 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 437b05806f564173921a9823c42e4239 in term 0.
I20260925 10:39:11.930828 25302 leader_election.cc:302] T f69bc748f3cb4ef79e77073b8e3c406f P 437b05806f564173921a9823c42e4239 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0311d5f20f3f449db871beefa5ae2330, 437b05806f564173921a9823c42e4239; no voters:
I20260925 10:39:11.931052 25604 raft_consensus.cc:2781] T f69bc748f3cb4ef79e77073b8e3c406f P 437b05806f564173921a9823c42e4239 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:39:11.931160 25604 raft_consensus.cc:482] T f69bc748f3cb4ef79e77073b8e3c406f P 437b05806f564173921a9823c42e4239 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:39:11.931255 25604 raft_consensus.cc:3037] T f69bc748f3cb4ef79e77073b8e3c406f P 437b05806f564173921a9823c42e4239 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:39:11.933427 25604 raft_consensus.cc:504] T f69bc748f3cb4ef79e77073b8e3c406f P 437b05806f564173921a9823c42e4239 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311d5f20f3f449db871beefa5ae2330" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 41861 } } peers { permanent_uuid: "437b05806f564173921a9823c42e4239" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 43899 } } peers { permanent_uuid: "499d386d8b3348f7924f6cf002a08253" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 39069 } }
I20260925 10:39:11.933933 25604 leader_election.cc:288] T f69bc748f3cb4ef79e77073b8e3c406f P 437b05806f564173921a9823c42e4239 [CANDIDATE]: Term 1 election: Requested vote from peers 0311d5f20f3f449db871beefa5ae2330 (127.23.100.1:41861), 499d386d8b3348f7924f6cf002a08253 (127.23.100.3:39069)
I20260925 10:39:11.934366 25236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69bc748f3cb4ef79e77073b8e3c406f" candidate_uuid: "437b05806f564173921a9823c42e4239" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0311d5f20f3f449db871beefa5ae2330"
I20260925 10:39:11.934464 25510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69bc748f3cb4ef79e77073b8e3c406f" candidate_uuid: "437b05806f564173921a9823c42e4239" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "499d386d8b3348f7924f6cf002a08253"
I20260925 10:39:11.934569 25236 raft_consensus.cc:3037] T f69bc748f3cb4ef79e77073b8e3c406f P 0311d5f20f3f449db871beefa5ae2330 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:39:11.934621 25510 raft_consensus.cc:3037] T f69bc748f3cb4ef79e77073b8e3c406f P 499d386d8b3348f7924f6cf002a08253 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:39:11.937270 25510 raft_consensus.cc:2454] T f69bc748f3cb4ef79e77073b8e3c406f P 499d386d8b3348f7924f6cf002a08253 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 437b05806f564173921a9823c42e4239 in term 1.
I20260925 10:39:11.937316 25236 raft_consensus.cc:2454] T f69bc748f3cb4ef79e77073b8e3c406f P 0311d5f20f3f449db871beefa5ae2330 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 437b05806f564173921a9823c42e4239 in term 1.
I20260925 10:39:11.937664 25302 leader_election.cc:302] T f69bc748f3cb4ef79e77073b8e3c406f P 437b05806f564173921a9823c42e4239 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0311d5f20f3f449db871beefa5ae2330, 437b05806f564173921a9823c42e4239; no voters:
I20260925 10:39:11.937950 25604 raft_consensus.cc:2781] T f69bc748f3cb4ef79e77073b8e3c406f P 437b05806f564173921a9823c42e4239 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:39:11.938107 25604 raft_consensus.cc:686] T f69bc748f3cb4ef79e77073b8e3c406f P 437b05806f564173921a9823c42e4239 [term 1 LEADER]: Becoming Leader. State: Replica: 437b05806f564173921a9823c42e4239, State: Running, Role: LEADER
I20260925 10:39:11.938476 25604 consensus_queue.cc:237] T f69bc748f3cb4ef79e77073b8e3c406f P 437b05806f564173921a9823c42e4239 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311d5f20f3f449db871beefa5ae2330" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 41861 } } peers { permanent_uuid: "437b05806f564173921a9823c42e4239" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 43899 } } peers { permanent_uuid: "499d386d8b3348f7924f6cf002a08253" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 39069 } }
I20260925 10:39:11.940850 25074 catalog_manager.cc:5495] T f69bc748f3cb4ef79e77073b8e3c406f P 437b05806f564173921a9823c42e4239 reported cstate change: term changed from 0 to 1, leader changed from <none> to 437b05806f564173921a9823c42e4239 (127.23.100.2). New cstate: current_term: 1 leader_uuid: "437b05806f564173921a9823c42e4239" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311d5f20f3f449db871beefa5ae2330" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 41861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "437b05806f564173921a9823c42e4239" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 43899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "499d386d8b3348f7924f6cf002a08253" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 39069 } health_report { overall_health: UNKNOWN } } }
Sep 25 10:39:11 dist-test-slave-dxdd krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332749, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.62@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:39:11.987638 25618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:11.972737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:37929 (local address 127.0.0.1:46382)
0925 10:39:11.973337 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:11.973364 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:11.973495 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:11.973951 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:11.973957 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:11.974838 (+ 881us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:11.975112 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:11.975123 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:11.976109 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:11.976115 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:11.977662 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:11.977676 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:11.978339 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:11.978346 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:11.978538 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:11.979222 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:11.979244 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:11.981722 (+ 2478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:11.985342 (+ 3620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:11.985351 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:11.985356 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:11.985784 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:11.986151 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:11.986154 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:11.986156 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:11.986291 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:11.986881 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:11.986887 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:11.987020 (+ 133us) client_negotiation.cc:769] Sending connection context
0925 10:39:11.987152 (+ 132us) client_negotiation.cc:240] Negotiation successful
0925 10:39:11.987326 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":297,"threads_started":1}
I20260925 10:39:11.988016 25619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:11.972764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46382 (local address 127.23.100.62:37929)
0925 10:39:11.973686 (+ 922us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:11.973690 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:11.973709 (+ 19us) server_negotiation.cc:407] Connection header received
0925 10:39:11.973749 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:11.973753 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:11.973802 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:11.973887 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:11.975249 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:11.975991 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:11.977786 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:11.978172 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:11.978259 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:11.981909 (+ 3650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:11.981916 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:11.981921 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:11.981956 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:11.985227 (+ 3271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:11.985887 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:11.985893 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:11.985897 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:11.986043 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:11.986414 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:11.986419 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:11.986423 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:11.986760 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:11.986912 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:11.987333 (+ 421us) server_negotiation.cc:299] Negotiation successful
0925 10:39:11.987560 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":833,"spinlock_wait_cycles":1920,"thread_start_us":721,"threads_started":1}
I20260925 10:39:11.993301 25074 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:46382:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 10:39:11.994604 25074 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:39:12.001346 25211 tablet_service.cc:1467] Processing CreateTablet for tablet 67930084b27d427ab67c05ea90cf20ad (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a58abbba6e54410ea7e75ab6e0d11a02]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 10:39:12.001355 25351 tablet_service.cc:1467] Processing CreateTablet for tablet 67930084b27d427ab67c05ea90cf20ad (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a58abbba6e54410ea7e75ab6e0d11a02]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 10:39:12.001855 25211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67930084b27d427ab67c05ea90cf20ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:39:12.001926 25351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67930084b27d427ab67c05ea90cf20ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:39:12.006944 25581 tablet_bootstrap.cc:492] T 67930084b27d427ab67c05ea90cf20ad P 0311d5f20f3f449db871beefa5ae2330: Bootstrap starting.
I20260925 10:39:12.007218 25490 tablet_service.cc:1467] Processing CreateTablet for tablet 67930084b27d427ab67c05ea90cf20ad (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a58abbba6e54410ea7e75ab6e0d11a02]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 10:39:12.007699 25490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67930084b27d427ab67c05ea90cf20ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:39:12.008585 25581 tablet_bootstrap.cc:654] T 67930084b27d427ab67c05ea90cf20ad P 0311d5f20f3f449db871beefa5ae2330: Neither blocks nor log segments found. Creating new log.
I20260925 10:39:12.010807 25581 tablet_bootstrap.cc:492] T 67930084b27d427ab67c05ea90cf20ad P 0311d5f20f3f449db871beefa5ae2330: No bootstrap required, opened a new log
I20260925 10:39:12.010965 25581 ts_tablet_manager.cc:1397] T 67930084b27d427ab67c05ea90cf20ad P 0311d5f20f3f449db871beefa5ae2330: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260925 10:39:12.011703 25581 raft_consensus.cc:348] T 67930084b27d427ab67c05ea90cf20ad P 0311d5f20f3f449db871beefa5ae2330 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311d5f20f3f449db871beefa5ae2330" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 41861 } } peers { permanent_uuid: "437b05806f564173921a9823c42e4239" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 43899 } } peers { permanent_uuid: "499d386d8b3348f7924f6cf002a08253" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 39069 } }
I20260925 10:39:12.011973 25581 raft_consensus.cc:374] T 67930084b27d427ab67c05ea90cf20ad P 0311d5f20f3f449db871beefa5ae2330 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:39:12.012055 25581 raft_consensus.cc:729] T 67930084b27d427ab67c05ea90cf20ad P 0311d5f20f3f449db871beefa5ae2330 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0311d5f20f3f449db871beefa5ae2330, State: Initialized, Role: FOLLOWER
I20260925 10:39:12.012336 25581 consensus_queue.cc:260] T 67930084b27d427ab67c05ea90cf20ad P 0311d5f20f3f449db871beefa5ae2330 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311d5f20f3f449db871beefa5ae2330" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 41861 } } peers { permanent_uuid: "437b05806f564173921a9823c42e4239" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 43899 } } peers { permanent_uuid: "499d386d8b3348f7924f6cf002a08253" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 39069 } }
I20260925 10:39:12.012768 25587 tablet_replica.cc:439] T 67930084b27d427ab67c05ea90cf20ad P 0311d5f20f3f449db871beefa5ae2330: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311d5f20f3f449db871beefa5ae2330" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 41861 } } peers { permanent_uuid: "437b05806f564173921a9823c42e4239" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 43899 } } peers { permanent_uuid: "499d386d8b3348f7924f6cf002a08253" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 39069 } } }
I20260925 10:39:12.013031 25587 tablet_replica.cc:442] T 67930084b27d427ab67c05ea90cf20ad P 0311d5f20f3f449db871beefa5ae2330: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:39:12.013566 25579 tablet_bootstrap.cc:492] T 67930084b27d427ab67c05ea90cf20ad P 499d386d8b3348f7924f6cf002a08253: Bootstrap starting.
I20260925 10:39:12.015271 25579 tablet_bootstrap.cc:654] T 67930084b27d427ab67c05ea90cf20ad P 499d386d8b3348f7924f6cf002a08253: Neither blocks nor log segments found. Creating new log.
I20260925 10:39:12.015290 25581 ts_tablet_manager.cc:1428] T 67930084b27d427ab67c05ea90cf20ad P 0311d5f20f3f449db871beefa5ae2330: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260925 10:39:12.018941 25580 tablet_bootstrap.cc:492] T 67930084b27d427ab67c05ea90cf20ad P 437b05806f564173921a9823c42e4239: Bootstrap starting.
I20260925 10:39:12.019292 25579 tablet_bootstrap.cc:492] T 67930084b27d427ab67c05ea90cf20ad P 499d386d8b3348f7924f6cf002a08253: No bootstrap required, opened a new log
I20260925 10:39:12.019454 25579 ts_tablet_manager.cc:1397] T 67930084b27d427ab67c05ea90cf20ad P 499d386d8b3348f7924f6cf002a08253: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 10:39:12.020226 25579 raft_consensus.cc:348] T 67930084b27d427ab67c05ea90cf20ad P 499d386d8b3348f7924f6cf002a08253 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311d5f20f3f449db871beefa5ae2330" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 41861 } } peers { permanent_uuid: "437b05806f564173921a9823c42e4239" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 43899 } } peers { permanent_uuid: "499d386d8b3348f7924f6cf002a08253" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 39069 } }
I20260925 10:39:12.020448 25579 raft_consensus.cc:374] T 67930084b27d427ab67c05ea90cf20ad P 499d386d8b3348f7924f6cf002a08253 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:39:12.020530 25579 raft_consensus.cc:729] T 67930084b27d427ab67c05ea90cf20ad P 499d386d8b3348f7924f6cf002a08253 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 499d386d8b3348f7924f6cf002a08253, State: Initialized, Role: FOLLOWER
I20260925 10:39:12.020681 25580 tablet_bootstrap.cc:654] T 67930084b27d427ab67c05ea90cf20ad P 437b05806f564173921a9823c42e4239: Neither blocks nor log segments found. Creating new log.
I20260925 10:39:12.020769 25579 consensus_queue.cc:260] T 67930084b27d427ab67c05ea90cf20ad P 499d386d8b3348f7924f6cf002a08253 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311d5f20f3f449db871beefa5ae2330" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 41861 } } peers { permanent_uuid: "437b05806f564173921a9823c42e4239" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 43899 } } peers { permanent_uuid: "499d386d8b3348f7924f6cf002a08253" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 39069 } }
I20260925 10:39:12.021360 25579 ts_tablet_manager.cc:1428] T 67930084b27d427ab67c05ea90cf20ad P 499d386d8b3348f7924f6cf002a08253: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:39:12.021237 25585 tablet_replica.cc:439] T 67930084b27d427ab67c05ea90cf20ad P 499d386d8b3348f7924f6cf002a08253: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311d5f20f3f449db871beefa5ae2330" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 41861 } } peers { permanent_uuid: "437b05806f564173921a9823c42e4239" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 43899 } } peers { permanent_uuid: "499d386d8b3348f7924f6cf002a08253" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 39069 } } }
I20260925 10:39:12.021600 25585 tablet_replica.cc:442] T 67930084b27d427ab67c05ea90cf20ad P 499d386d8b3348f7924f6cf002a08253: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:39:12.023357 25580 tablet_bootstrap.cc:492] T 67930084b27d427ab67c05ea90cf20ad P 437b05806f564173921a9823c42e4239: No bootstrap required, opened a new log
I20260925 10:39:12.023555 25580 ts_tablet_manager.cc:1397] T 67930084b27d427ab67c05ea90cf20ad P 437b05806f564173921a9823c42e4239: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260925 10:39:12.024202 25580 raft_consensus.cc:348] T 67930084b27d427ab67c05ea90cf20ad P 437b05806f564173921a9823c42e4239 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311d5f20f3f449db871beefa5ae2330" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 41861 } } peers { permanent_uuid: "437b05806f564173921a9823c42e4239" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 43899 } } peers { permanent_uuid: "499d386d8b3348f7924f6cf002a08253" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 39069 } }
I20260925 10:39:12.024408 25580 raft_consensus.cc:374] T 67930084b27d427ab67c05ea90cf20ad P 437b05806f564173921a9823c42e4239 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:39:12.024492 25580 raft_consensus.cc:729] T 67930084b27d427ab67c05ea90cf20ad P 437b05806f564173921a9823c42e4239 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 437b05806f564173921a9823c42e4239, State: Initialized, Role: FOLLOWER
I20260925 10:39:12.024731 25580 consensus_queue.cc:260] T 67930084b27d427ab67c05ea90cf20ad P 437b05806f564173921a9823c42e4239 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311d5f20f3f449db871beefa5ae2330" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 41861 } } peers { permanent_uuid: "437b05806f564173921a9823c42e4239" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 43899 } } peers { permanent_uuid: "499d386d8b3348f7924f6cf002a08253" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 39069 } }
I20260925 10:39:12.025146 25604 tablet_replica.cc:439] T 67930084b27d427ab67c05ea90cf20ad P 437b05806f564173921a9823c42e4239: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311d5f20f3f449db871beefa5ae2330" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 41861 } } peers { permanent_uuid: "437b05806f564173921a9823c42e4239" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 43899 } } peers { permanent_uuid: "499d386d8b3348f7924f6cf002a08253" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 39069 } } }
I20260925 10:39:12.025363 25580 ts_tablet_manager.cc:1428] T 67930084b27d427ab67c05ea90cf20ad P 437b05806f564173921a9823c42e4239: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260925 10:39:12.025439 25604 tablet_replica.cc:442] T 67930084b27d427ab67c05ea90cf20ad P 437b05806f564173921a9823c42e4239: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:39:12.358263 25587 raft_consensus.cc:482] T 67930084b27d427ab67c05ea90cf20ad P 0311d5f20f3f449db871beefa5ae2330 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:39:12.358501 25587 raft_consensus.cc:504] T 67930084b27d427ab67c05ea90cf20ad P 0311d5f20f3f449db871beefa5ae2330 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311d5f20f3f449db871beefa5ae2330" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 41861 } } peers { permanent_uuid: "437b05806f564173921a9823c42e4239" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 43899 } } peers { permanent_uuid: "499d386d8b3348f7924f6cf002a08253" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 39069 } }
I20260925 10:39:12.359226 25587 leader_election.cc:288] T 67930084b27d427ab67c05ea90cf20ad P 0311d5f20f3f449db871beefa5ae2330 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 437b05806f564173921a9823c42e4239 (127.23.100.2:43899), 499d386d8b3348f7924f6cf002a08253 (127.23.100.3:39069)
I20260925 10:39:12.359772 25376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67930084b27d427ab67c05ea90cf20ad" candidate_uuid: "0311d5f20f3f449db871beefa5ae2330" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "437b05806f564173921a9823c42e4239" is_pre_election: true
I20260925 10:39:12.359902 25510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67930084b27d427ab67c05ea90cf20ad" candidate_uuid: "0311d5f20f3f449db871beefa5ae2330" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "499d386d8b3348f7924f6cf002a08253" is_pre_election: true
I20260925 10:39:12.360028 25376 raft_consensus.cc:2454] T 67930084b27d427ab67c05ea90cf20ad P 437b05806f564173921a9823c42e4239 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0311d5f20f3f449db871beefa5ae2330 in term 0.
I20260925 10:39:12.360163 25510 raft_consensus.cc:2454] T 67930084b27d427ab67c05ea90cf20ad P 499d386d8b3348f7924f6cf002a08253 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0311d5f20f3f449db871beefa5ae2330 in term 0.
I20260925 10:39:12.360455 25162 leader_election.cc:302] T 67930084b27d427ab67c05ea90cf20ad P 0311d5f20f3f449db871beefa5ae2330 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0311d5f20f3f449db871beefa5ae2330, 437b05806f564173921a9823c42e4239; no voters:
I20260925 10:39:12.360687 25587 raft_consensus.cc:2781] T 67930084b27d427ab67c05ea90cf20ad P 0311d5f20f3f449db871beefa5ae2330 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:39:12.360805 25587 raft_consensus.cc:482] T 67930084b27d427ab67c05ea90cf20ad P 0311d5f20f3f449db871beefa5ae2330 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:39:12.360906 25587 raft_consensus.cc:3037] T 67930084b27d427ab67c05ea90cf20ad P 0311d5f20f3f449db871beefa5ae2330 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:39:12.362955 25587 raft_consensus.cc:504] T 67930084b27d427ab67c05ea90cf20ad P 0311d5f20f3f449db871beefa5ae2330 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311d5f20f3f449db871beefa5ae2330" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 41861 } } peers { permanent_uuid: "437b05806f564173921a9823c42e4239" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 43899 } } peers { permanent_uuid: "499d386d8b3348f7924f6cf002a08253" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 39069 } }
I20260925 10:39:12.363507 25587 leader_election.cc:288] T 67930084b27d427ab67c05ea90cf20ad P 0311d5f20f3f449db871beefa5ae2330 [CANDIDATE]: Term 1 election: Requested vote from peers 437b05806f564173921a9823c42e4239 (127.23.100.2:43899), 499d386d8b3348f7924f6cf002a08253 (127.23.100.3:39069)
I20260925 10:39:12.363972 25376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67930084b27d427ab67c05ea90cf20ad" candidate_uuid: "0311d5f20f3f449db871beefa5ae2330" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "437b05806f564173921a9823c42e4239"
I20260925 10:39:12.364058 25510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67930084b27d427ab67c05ea90cf20ad" candidate_uuid: "0311d5f20f3f449db871beefa5ae2330" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "499d386d8b3348f7924f6cf002a08253"
I20260925 10:39:12.364185 25376 raft_consensus.cc:3037] T 67930084b27d427ab67c05ea90cf20ad P 437b05806f564173921a9823c42e4239 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:39:12.364287 25510 raft_consensus.cc:3037] T 67930084b27d427ab67c05ea90cf20ad P 499d386d8b3348f7924f6cf002a08253 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:39:12.366456 25376 raft_consensus.cc:2454] T 67930084b27d427ab67c05ea90cf20ad P 437b05806f564173921a9823c42e4239 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0311d5f20f3f449db871beefa5ae2330 in term 1.
I20260925 10:39:12.366811 25162 leader_election.cc:302] T 67930084b27d427ab67c05ea90cf20ad P 0311d5f20f3f449db871beefa5ae2330 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0311d5f20f3f449db871beefa5ae2330, 437b05806f564173921a9823c42e4239; no voters:
I20260925 10:39:12.367075 25587 raft_consensus.cc:2781] T 67930084b27d427ab67c05ea90cf20ad P 0311d5f20f3f449db871beefa5ae2330 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:39:12.367211 25510 raft_consensus.cc:2454] T 67930084b27d427ab67c05ea90cf20ad P 499d386d8b3348f7924f6cf002a08253 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0311d5f20f3f449db871beefa5ae2330 in term 1.
I20260925 10:39:12.367605 25587 raft_consensus.cc:686] T 67930084b27d427ab67c05ea90cf20ad P 0311d5f20f3f449db871beefa5ae2330 [term 1 LEADER]: Becoming Leader. State: Replica: 0311d5f20f3f449db871beefa5ae2330, State: Running, Role: LEADER
I20260925 10:39:12.368038 25587 consensus_queue.cc:237] T 67930084b27d427ab67c05ea90cf20ad P 0311d5f20f3f449db871beefa5ae2330 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311d5f20f3f449db871beefa5ae2330" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 41861 } } peers { permanent_uuid: "437b05806f564173921a9823c42e4239" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 43899 } } peers { permanent_uuid: "499d386d8b3348f7924f6cf002a08253" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 39069 } }
I20260925 10:39:12.369833 25623 tablet_replica.cc:439] T 67930084b27d427ab67c05ea90cf20ad P 0311d5f20f3f449db871beefa5ae2330: TxnStatusTablet state changed. Reason: New leader 0311d5f20f3f449db871beefa5ae2330. Latest consensus state: current_term: 1 leader_uuid: "0311d5f20f3f449db871beefa5ae2330" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311d5f20f3f449db871beefa5ae2330" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 41861 } } peers { permanent_uuid: "437b05806f564173921a9823c42e4239" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 43899 } } peers { permanent_uuid: "499d386d8b3348f7924f6cf002a08253" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 39069 } } }
I20260925 10:39:12.370232 25623 tablet_replica.cc:442] T 67930084b27d427ab67c05ea90cf20ad P 0311d5f20f3f449db871beefa5ae2330: This TxnStatusTablet replica's current role is: LEADER
I20260925 10:39:12.370852 25625 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 10:39:12.371063 25625 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 10:39:12.371685 25074 catalog_manager.cc:5495] T 67930084b27d427ab67c05ea90cf20ad P 0311d5f20f3f449db871beefa5ae2330 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0311d5f20f3f449db871beefa5ae2330 (127.23.100.1). New cstate: current_term: 1 leader_uuid: "0311d5f20f3f449db871beefa5ae2330" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311d5f20f3f449db871beefa5ae2330" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 41861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "437b05806f564173921a9823c42e4239" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 43899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "499d386d8b3348f7924f6cf002a08253" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 39069 } health_report { overall_health: UNKNOWN } } }
I20260925 10:39:12.394093 25628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:12.385959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49150 (local address 127.23.100.1:41861)
0925 10:39:12.386903 (+ 944us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:12.386908 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:12.386938 (+ 30us) server_negotiation.cc:407] Connection header received
0925 10:39:12.387025 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:12.387035 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:12.387128 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:12.387270 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:39:12.387708 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:12.389029 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:12.391032 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:12.391806 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:12.391973 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:12.392596 (+ 623us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:39:12.393372 (+ 776us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:39:12.393501 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:12.393643 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 10:39:12.393725 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":796,"thread_start_us":645,"threads_started":1}
I20260925 10:39:12.394084 25618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:12.385843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:41861 (local address 127.0.0.1:49150)
0925 10:39:12.386157 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:12.386175 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:12.386293 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:12.387321 (+ 1028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:12.387326 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:12.387340 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:39:12.387528 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:12.387539 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:12.389181 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:12.389187 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:12.390863 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:12.390878 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:12.391987 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:12.391995 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:12.392242 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:12.392315 (+ 73us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:39:12.393485 (+ 1170us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:39:12.393508 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 10:39:12.393603 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 10:39:12.393770 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164}
I20260925 10:39:12.398891 25510 raft_consensus.cc:1261] T 67930084b27d427ab67c05ea90cf20ad P 499d386d8b3348f7924f6cf002a08253 [term 1 FOLLOWER]: Refusing update from remote peer 0311d5f20f3f449db871beefa5ae2330: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:39:12.399063 25376 raft_consensus.cc:1261] T 67930084b27d427ab67c05ea90cf20ad P 437b05806f564173921a9823c42e4239 [term 1 FOLLOWER]: Refusing update from remote peer 0311d5f20f3f449db871beefa5ae2330: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:39:12.399690 25623 consensus_queue.cc:1035] T 67930084b27d427ab67c05ea90cf20ad P 0311d5f20f3f449db871beefa5ae2330 [LEADER]: Connected to new peer: Peer: permanent_uuid: "499d386d8b3348f7924f6cf002a08253" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 39069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:39:12.400029 25587 consensus_queue.cc:1035] T 67930084b27d427ab67c05ea90cf20ad P 0311d5f20f3f449db871beefa5ae2330 [LEADER]: Connected to new peer: Peer: permanent_uuid: "437b05806f564173921a9823c42e4239" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 43899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:39:12.418742 25585 tablet_replica.cc:439] T 67930084b27d427ab67c05ea90cf20ad P 499d386d8b3348f7924f6cf002a08253: TxnStatusTablet state changed. Reason: New leader 0311d5f20f3f449db871beefa5ae2330. Latest consensus state: current_term: 1 leader_uuid: "0311d5f20f3f449db871beefa5ae2330" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311d5f20f3f449db871beefa5ae2330" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 41861 } } peers { permanent_uuid: "437b05806f564173921a9823c42e4239" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 43899 } } peers { permanent_uuid: "499d386d8b3348f7924f6cf002a08253" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 39069 } } }
I20260925 10:39:12.419034 25585 tablet_replica.cc:442] T 67930084b27d427ab67c05ea90cf20ad P 499d386d8b3348f7924f6cf002a08253: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:39:12.420861 25604 tablet_replica.cc:439] T 67930084b27d427ab67c05ea90cf20ad P 437b05806f564173921a9823c42e4239: TxnStatusTablet state changed. Reason: New leader 0311d5f20f3f449db871beefa5ae2330. Latest consensus state: current_term: 1 leader_uuid: "0311d5f20f3f449db871beefa5ae2330" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311d5f20f3f449db871beefa5ae2330" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 41861 } } peers { permanent_uuid: "437b05806f564173921a9823c42e4239" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 43899 } } peers { permanent_uuid: "499d386d8b3348f7924f6cf002a08253" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 39069 } } }
I20260925 10:39:12.421132 25604 tablet_replica.cc:442] T 67930084b27d427ab67c05ea90cf20ad P 437b05806f564173921a9823c42e4239: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:39:12.421240 25629 mvcc.cc:205] Tried to move back new op lower bound from 7333202953818075136 to 7333202953702940672. Current Snapshot: MvccSnapshot[applied={T|T < 7333202953818075136}]
I20260925 10:39:12.426451 25604 tablet_replica.cc:439] T 67930084b27d427ab67c05ea90cf20ad P 437b05806f564173921a9823c42e4239: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0311d5f20f3f449db871beefa5ae2330" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311d5f20f3f449db871beefa5ae2330" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 41861 } } peers { permanent_uuid: "437b05806f564173921a9823c42e4239" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 43899 } } peers { permanent_uuid: "499d386d8b3348f7924f6cf002a08253" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 39069 } } }
I20260925 10:39:12.426777 25604 tablet_replica.cc:442] T 67930084b27d427ab67c05ea90cf20ad P 437b05806f564173921a9823c42e4239: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:39:12.427943 25587 tablet_replica.cc:439] T 67930084b27d427ab67c05ea90cf20ad P 0311d5f20f3f449db871beefa5ae2330: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0311d5f20f3f449db871beefa5ae2330" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311d5f20f3f449db871beefa5ae2330" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 41861 } } peers { permanent_uuid: "437b05806f564173921a9823c42e4239" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 43899 } } peers { permanent_uuid: "499d386d8b3348f7924f6cf002a08253" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 39069 } } }
I20260925 10:39:12.428304 25587 tablet_replica.cc:442] T 67930084b27d427ab67c05ea90cf20ad P 0311d5f20f3f449db871beefa5ae2330: This TxnStatusTablet replica's current role is: LEADER
I20260925 10:39:12.431221 25585 tablet_replica.cc:439] T 67930084b27d427ab67c05ea90cf20ad P 499d386d8b3348f7924f6cf002a08253: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0311d5f20f3f449db871beefa5ae2330" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311d5f20f3f449db871beefa5ae2330" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 41861 } } peers { permanent_uuid: "437b05806f564173921a9823c42e4239" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 43899 } } peers { permanent_uuid: "499d386d8b3348f7924f6cf002a08253" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 39069 } } }
I20260925 10:39:12.431532 25585 tablet_replica.cc:442] T 67930084b27d427ab67c05ea90cf20ad P 499d386d8b3348f7924f6cf002a08253: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:39:12.435094 25604 consensus_queue.cc:1035] T f69bc748f3cb4ef79e77073b8e3c406f P 437b05806f564173921a9823c42e4239 [LEADER]: Connected to new peer: Peer: permanent_uuid: "499d386d8b3348f7924f6cf002a08253" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 39069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:39:12.444298 25623 tablet_replica.cc:439] T 67930084b27d427ab67c05ea90cf20ad P 0311d5f20f3f449db871beefa5ae2330: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0311d5f20f3f449db871beefa5ae2330" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311d5f20f3f449db871beefa5ae2330" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 41861 } } peers { permanent_uuid: "437b05806f564173921a9823c42e4239" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 43899 } } peers { permanent_uuid: "499d386d8b3348f7924f6cf002a08253" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 39069 } } }
I20260925 10:39:12.444723 25623 tablet_replica.cc:442] T 67930084b27d427ab67c05ea90cf20ad P 0311d5f20f3f449db871beefa5ae2330: This TxnStatusTablet replica's current role is: LEADER
I20260925 10:39:12.463634 25604 consensus_queue.cc:1035] T f69bc748f3cb4ef79e77073b8e3c406f P 437b05806f564173921a9823c42e4239 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0311d5f20f3f449db871beefa5ae2330" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 41861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:39:12.472108 25628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:12.464875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:33859 (local address 127.23.100.1:41861)
0925 10:39:12.465053 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:12.465061 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:12.466110 (+ 1049us) server_negotiation.cc:407] Connection header received
0925 10:39:12.466345 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:12.466351 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:12.466437 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:12.466537 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:39:12.467169 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:12.468340 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:12.470498 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:12.470926 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:12.471022 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:12.471552 (+ 530us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:39:12.471640 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:39:12.471705 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:12.471845 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 10:39:12.471907 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 10:39:12.472375 25644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:12.465311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:41861 (local address 127.23.100.2:33859)
0925 10:39:12.465982 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:12.466007 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:12.466187 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:12.466731 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:12.466739 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:12.466756 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:39:12.466990 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:12.467004 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:12.468501 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:12.468509 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:12.470137 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:12.470155 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:12.471110 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:12.471118 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:12.471299 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:12.471374 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:39:12.471733 (+ 359us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:39:12.471746 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 10:39:12.471829 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 10:39:12.471987 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":309,"threads_started":1}
I20260925 10:39:12.513479 25648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:12.504185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.2:43899 (local address 127.23.100.1:59801)
0925 10:39:12.505253 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:12.505279 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:12.505468 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:12.507285 (+ 1817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:12.507290 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:12.507304 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:39:12.507745 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:12.507761 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:12.509101 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:12.509107 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:12.511004 (+ 1897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:12.511020 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:12.512080 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:12.512088 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:12.512270 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:12.512345 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:39:12.512870 (+ 525us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:39:12.512880 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 10:39:12.512953 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 10:39:12.513057 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":823,"thread_start_us":603,"threads_started":1}
I20260925 10:39:12.513487 25649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:12.504523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:59801 (local address 127.23.100.2:43899)
0925 10:39:12.506972 (+ 2449us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:12.506978 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:12.507003 (+ 25us) server_negotiation.cc:407] Connection header received
0925 10:39:12.507064 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:12.507069 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:12.507147 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:12.507251 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:39:12.507958 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:12.508969 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:12.511203 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:12.511867 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:12.512515 (+ 648us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:12.512649 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:39:12.512776 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:39:12.512866 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:12.512996 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 10:39:12.513078 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2323,"thread_start_us":2210,"threads_started":1}
I20260925 10:39:12.545646 25073 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:39:12.545858 25073 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:39:12.548651 25073 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 17626f350f1e4739a7e884a2be1a8823: Sending DeleteTablet for 3 replicas of tablet f69bc748f3cb4ef79e77073b8e3c406f
I20260925 10:39:12.549355 25211 tablet_service.cc:1514] Processing DeleteTablet for tablet f69bc748f3cb4ef79e77073b8e3c406f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:39:12 UTC) from {username='kudu', principal='kudu/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:49142
I20260925 10:39:12.549575 25490 tablet_service.cc:1514] Processing DeleteTablet for tablet f69bc748f3cb4ef79e77073b8e3c406f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:39:12 UTC) from {username='kudu', principal='kudu/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:53470
I20260925 10:39:12.549817 25351 tablet_service.cc:1514] Processing DeleteTablet for tablet f69bc748f3cb4ef79e77073b8e3c406f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:39:12 UTC) from {username='kudu', principal='kudu/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:42406
I20260925 10:39:12.550148 25652 tablet_replica.cc:330] T f69bc748f3cb4ef79e77073b8e3c406f P 499d386d8b3348f7924f6cf002a08253: stopping tablet replica
I20260925 10:39:12.550431 25653 tablet_replica.cc:330] T f69bc748f3cb4ef79e77073b8e3c406f P 437b05806f564173921a9823c42e4239: stopping tablet replica
I20260925 10:39:12.550450 25652 raft_consensus.cc:2229] T f69bc748f3cb4ef79e77073b8e3c406f P 499d386d8b3348f7924f6cf002a08253 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:39:12.550662 25653 raft_consensus.cc:2229] T f69bc748f3cb4ef79e77073b8e3c406f P 437b05806f564173921a9823c42e4239 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:39:12.550698 25652 raft_consensus.cc:2258] T f69bc748f3cb4ef79e77073b8e3c406f P 499d386d8b3348f7924f6cf002a08253 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:39:12.551017 25653 raft_consensus.cc:2258] T f69bc748f3cb4ef79e77073b8e3c406f P 437b05806f564173921a9823c42e4239 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:39:12.551748 25652 ts_tablet_manager.cc:1905] T f69bc748f3cb4ef79e77073b8e3c406f P 499d386d8b3348f7924f6cf002a08253: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:39:12.552135 25653 ts_tablet_manager.cc:1905] T f69bc748f3cb4ef79e77073b8e3c406f P 437b05806f564173921a9823c42e4239: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:39:12.558740 25652 ts_tablet_manager.cc:1918] T f69bc748f3cb4ef79e77073b8e3c406f P 499d386d8b3348f7924f6cf002a08253: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:39:12.558946 25652 log.cc:1196] T f69bc748f3cb4ef79e77073b8e3c406f P 499d386d8b3348f7924f6cf002a08253: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/ts-2/wal/wals/f69bc748f3cb4ef79e77073b8e3c406f
I20260925 10:39:12.559473 25652 ts_tablet_manager.cc:1939] T f69bc748f3cb4ef79e77073b8e3c406f P 499d386d8b3348f7924f6cf002a08253: Deleting consensus metadata
I20260925 10:39:12.560578 25059 catalog_manager.cc:4841] TS 499d386d8b3348f7924f6cf002a08253 (127.23.100.3:39069): tablet f69bc748f3cb4ef79e77073b8e3c406f (table test-table [id=24964006bfb947558428d4b81dc3a617]) successfully deleted
I20260925 10:39:12.560796 25653 ts_tablet_manager.cc:1918] T f69bc748f3cb4ef79e77073b8e3c406f P 437b05806f564173921a9823c42e4239: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:39:12.561018 25653 log.cc:1196] T f69bc748f3cb4ef79e77073b8e3c406f P 437b05806f564173921a9823c42e4239: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/ts-1/wal/wals/f69bc748f3cb4ef79e77073b8e3c406f
I20260925 10:39:12.561584 25653 ts_tablet_manager.cc:1939] T f69bc748f3cb4ef79e77073b8e3c406f P 437b05806f564173921a9823c42e4239: Deleting consensus metadata
I20260925 10:39:12.562745 25059 catalog_manager.cc:4841] TS 437b05806f564173921a9823c42e4239 (127.23.100.2:43899): tablet f69bc748f3cb4ef79e77073b8e3c406f (table test-table [id=24964006bfb947558428d4b81dc3a617]) successfully deleted
I20260925 10:39:12.562938 25651 tablet_replica.cc:330] T f69bc748f3cb4ef79e77073b8e3c406f P 0311d5f20f3f449db871beefa5ae2330: stopping tablet replica
I20260925 10:39:12.563190 25651 raft_consensus.cc:2229] T f69bc748f3cb4ef79e77073b8e3c406f P 0311d5f20f3f449db871beefa5ae2330 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:39:12.563544 25651 raft_consensus.cc:2258] T f69bc748f3cb4ef79e77073b8e3c406f P 0311d5f20f3f449db871beefa5ae2330 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:39:12.564678 25651 ts_tablet_manager.cc:1905] T f69bc748f3cb4ef79e77073b8e3c406f P 0311d5f20f3f449db871beefa5ae2330: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:39:12.571880 25651 ts_tablet_manager.cc:1918] T f69bc748f3cb4ef79e77073b8e3c406f P 0311d5f20f3f449db871beefa5ae2330: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:39:12.572029 25651 log.cc:1196] T f69bc748f3cb4ef79e77073b8e3c406f P 0311d5f20f3f449db871beefa5ae2330: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790332740062128-23952-0/minicluster-data/ts-0/wal/wals/f69bc748f3cb4ef79e77073b8e3c406f
I20260925 10:39:12.572489 25651 ts_tablet_manager.cc:1939] T f69bc748f3cb4ef79e77073b8e3c406f P 0311d5f20f3f449db871beefa5ae2330: Deleting consensus metadata
I20260925 10:39:12.573544 25059 catalog_manager.cc:4841] TS 0311d5f20f3f449db871beefa5ae2330 (127.23.100.1:41861): tablet f69bc748f3cb4ef79e77073b8e3c406f (table test-table [id=24964006bfb947558428d4b81dc3a617]) successfully deleted
Sep 25 10:39:12 dist-test-slave-dxdd krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332751, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.100.1@KRBTEST.COM
I20260925 10:39:12.585641 25628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:12.563465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49158 (local address 127.23.100.1:41861)
0925 10:39:12.563712 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:12.563721 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:12.570827 (+ 7106us) server_negotiation.cc:407] Connection header received
0925 10:39:12.571051 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:12.571054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:12.571119 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:12.571225 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:12.572757 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:12.573706 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:12.574977 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:12.575413 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:12.575547 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:12.579732 (+ 4185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:12.579742 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:12.579757 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:12.579797 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:12.582910 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:12.583720 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:12.583726 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:12.583729 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:12.583858 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:12.584231 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:12.584235 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:12.584237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:12.584561 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:12.584783 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:12.585144 (+ 361us) server_negotiation.cc:299] Negotiation successful
0925 10:39:12.585298 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260925 10:39:12.586413 25277 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49158
I20260925 10:39:12.601424 25658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:12.589163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46388 (local address 127.23.100.62:37929)
0925 10:39:12.589802 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:12.589807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:12.589821 (+ 14us) server_negotiation.cc:407] Connection header received
0925 10:39:12.589864 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:12.589867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:12.589920 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:12.590029 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:12.591278 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:12.592186 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:12.593473 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:12.593885 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:12.594013 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:12.596199 (+ 2186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:12.596204 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:12.596208 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:12.596251 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:12.598993 (+ 2742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:12.599585 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:12.599591 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:12.599595 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:12.599709 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:12.600048 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:12.600053 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:12.600057 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:12.600327 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:12.600579 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:12.600896 (+ 317us) server_negotiation.cc:299] Negotiation successful
0925 10:39:12.601102 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":341,"threads_started":1}
W20260925 10:39:12.601980 25073 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46388
I20260925 10:39:12.603706 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 25146
I20260925 10:39:12.618268 25628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:12.604784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49168 (local address 127.23.100.1:41861)
0925 10:39:12.605013 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:12.605019 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:12.605399 (+ 380us) server_negotiation.cc:407] Connection header received
0925 10:39:12.605503 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:12.605508 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:12.605622 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:12.605717 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:12.606827 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:12.607884 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:12.608994 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:12.609661 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:12.609791 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:12.611937 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:12.611945 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:12.611955 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:12.611993 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:12.615702 (+ 3709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:12.616277 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:12.616282 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:12.616286 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:12.616439 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:12.616788 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:12.616793 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:12.616796 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:12.617174 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:12.617484 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:12.617623 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 10:39:12.617857 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260925 10:39:12.618876 25277 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49168
W20260925 10:39:12.619410 23952 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 25146: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:39:12.619599 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 25146
I20260925 10:39:12.639004 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 25286
Sep 25 10:39:12 dist-test-slave-dxdd krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332751, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.100.2@KRBTEST.COM
I20260925 10:39:12.652904 25649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:12.639766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42432 (local address 127.23.100.2:43899)
0925 10:39:12.639940 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:12.639944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:12.639961 (+ 17us) server_negotiation.cc:407] Connection header received
0925 10:39:12.640033 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:12.640037 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:12.640099 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:12.640183 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:12.641391 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:12.642137 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:12.643440 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:12.643871 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:12.643964 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:12.647878 (+ 3914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:12.647890 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:12.647904 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:12.647947 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:12.650684 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:12.651241 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:12.651247 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:12.651249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:12.651375 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:12.651657 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:12.651660 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:12.651662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:12.651959 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:12.652153 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:12.652380 (+ 227us) server_negotiation.cc:299] Negotiation successful
0925 10:39:12.652526 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260925 10:39:12.653537 25416 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42432
W20260925 10:39:12.654055 23952 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 25286: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:39:12.654187 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 25286
I20260925 10:39:12.674883 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 25425
Sep 25 10:39:12 dist-test-slave-dxdd krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332751, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.100.3@KRBTEST.COM
I20260925 10:39:12.689110 25660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:12.675811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53486 (local address 127.23.100.3:39069)
0925 10:39:12.676312 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:12.676317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:12.676332 (+ 15us) server_negotiation.cc:407] Connection header received
0925 10:39:12.676381 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:12.676385 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:12.676439 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:12.676528 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:12.677617 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:12.678408 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:12.679571 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:12.679989 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:12.680075 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:12.683973 (+ 3898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:12.683986 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:12.684001 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:12.684030 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:12.686844 (+ 2814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:12.687416 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:12.687422 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:12.687424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:12.687556 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:12.687854 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:12.687857 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:12.687859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:12.688167 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:12.688406 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:12.688584 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 10:39:12.688732 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":280,"threads_started":1}
W20260925 10:39:12.689790 25555 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53486
W20260925 10:39:12.690424 23952 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 25425: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:39:12.690574 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 25425
I20260925 10:39:12.706746 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 25043
I20260925 10:39:12.707919 25115 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:39:12.819774 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 25043
2026-09-25T10:39:12Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4690 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:39:12 dist-test-slave-dxdd krb5kdc[25664](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:39:12 dist-test-slave-dxdd krb5kdc[25664](info): set up 2 sockets
Sep 25 10:39:12 dist-test-slave-dxdd krb5kdc[25664](info): commencing operation
krb5kdc: starting...
W20260925 10:39:13.902451 23952 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.007s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Sep 25 10:39:14 dist-test-slave-dxdd krb5kdc[25664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:39:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:39:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
I20260925 10:39:14.079823 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:35469
--webserver_interface=127.23.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:39451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.62:35469
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:39:14.408250 25677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:39:14.408521 25677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:39:14.408685 25677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:39:14.415582 25677 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:39:14.415683 25677 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 10:39:14.415771 25677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:39:14.415916 25677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:39:14.416004 25677 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:39:14.416109 25677 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:39:14.425493 25677 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:39451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.62:35469
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:35469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:39:14.426707 25677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:39:14.427979 25677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:39:14.439385 25682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:14.439342 25683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:14.439424 25685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:14.440143 25677 server_base.cc:1034] running on GCE node
I20260925 10:39:14.440865 25677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:39:14.442159 25677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:39:14.443394 25677 hybrid_clock.cc:648] HybridClock initialized: now 1790332754443339 us; error 57 us; skew 500 ppm
I20260925 10:39:14.443715 25677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:39:14 dist-test-slave-dxdd krb5kdc[25664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332754, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:39:14.446663 25677 init.cc:377] Logged in from keytab as kudu/127.23.100.62@KRBTEST.COM (short username kudu)
I20260925 10:39:14.448047 25677 webserver.cc:460] Webserver started at http://127.23.100.62:42201/ using document root <none> and password file <none>
I20260925 10:39:14.448728 25677 fs_manager.cc:359] Metadata directory not provided
I20260925 10:39:14.448855 25677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:39:14.449160 25677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:39:14.452008 25677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/master-0/data/instance:
uuid: "beeffe7071f041e297918aa03aa556c9"
format_stamp: "Formatted at 2026-09-25 10:39:14 on dist-test-slave-dxdd"
server_key: "a9a2a88c1fd3962b0d2ca7e443d37c49"
server_key_iv: "11d89308b2159af94068aff2b55ce916"
server_key_version: "encryptionkey@0"
I20260925 10:39:14.452780 25677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/master-0/wal/instance:
uuid: "beeffe7071f041e297918aa03aa556c9"
format_stamp: "Formatted at 2026-09-25 10:39:14 on dist-test-slave-dxdd"
server_key: "a9a2a88c1fd3962b0d2ca7e443d37c49"
server_key_iv: "11d89308b2159af94068aff2b55ce916"
server_key_version: "encryptionkey@0"
I20260925 10:39:14.457647 25677 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 10:39:14.460875 25692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:14.462102 25677 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 10:39:14.462270 25677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/master-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/master-0/wal
uuid: "beeffe7071f041e297918aa03aa556c9"
format_stamp: "Formatted at 2026-09-25 10:39:14 on dist-test-slave-dxdd"
server_key: "a9a2a88c1fd3962b0d2ca7e443d37c49"
server_key_iv: "11d89308b2159af94068aff2b55ce916"
server_key_version: "encryptionkey@0"
I20260925 10:39:14.462556 25677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:39:14.477205 25677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:39:14.481617 25677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:39:14.481873 25677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:39:14.503539 25677 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.62:35469
I20260925 10:39:14.503551 25753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.62:35469 every 8 connection(s)
I20260925 10:39:14.504868 25677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
I20260925 10:39:14.506656 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 25677
I20260925 10:39:14.506866 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/master-0/wal/instance
I20260925 10:39:14.507649 23952 external_mini_cluster.cc:1428] Setting key 838882a635f9bc0127068dce69f95663
I20260925 10:39:14.509287 25754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 10:39:14 dist-test-slave-dxdd krb5kdc[25664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:39:14.522413 25754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P beeffe7071f041e297918aa03aa556c9: Bootstrap starting.
I20260925 10:39:14.526348 25754 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P beeffe7071f041e297918aa03aa556c9: Neither blocks nor log segments found. Creating new log.
I20260925 10:39:14.527575 25754 log.cc:824] T 00000000000000000000000000000000 P beeffe7071f041e297918aa03aa556c9: Log is configured to *not* fsync() on all Append() calls
I20260925 10:39:14.531108 25754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P beeffe7071f041e297918aa03aa556c9: No bootstrap required, opened a new log
I20260925 10:39:14.531211 25757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:14.509695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43766 (local address 127.23.100.62:35469)
0925 10:39:14.510488 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:14.510502 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:14.510551 (+ 49us) server_negotiation.cc:407] Connection header received
0925 10:39:14.512431 (+ 1880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:14.512456 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:14.512944 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:14.513367 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:14.514643 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:14.516654 (+ 2011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:14.517897 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:14.518908 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:14.519122 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:14.522804 (+ 3682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:14.522815 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:14.522830 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:14.522876 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:14.526813 (+ 3937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:14.527550 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:14.527565 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:14.527574 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:14.527773 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:14.528194 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:14.528202 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:14.528208 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:14.528715 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:14.528945 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:14.529519 (+ 574us) server_negotiation.cc:299] Negotiation successful
0925 10:39:14.529920 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":326,"threads_started":1}
I20260925 10:39:14.538672 25754 raft_consensus.cc:348] T 00000000000000000000000000000000 P beeffe7071f041e297918aa03aa556c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beeffe7071f041e297918aa03aa556c9" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 35469 } }
I20260925 10:39:14.538973 25754 raft_consensus.cc:374] T 00000000000000000000000000000000 P beeffe7071f041e297918aa03aa556c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:39:14.539059 25754 raft_consensus.cc:729] T 00000000000000000000000000000000 P beeffe7071f041e297918aa03aa556c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beeffe7071f041e297918aa03aa556c9, State: Initialized, Role: FOLLOWER
I20260925 10:39:14.539561 25754 consensus_queue.cc:260] T 00000000000000000000000000000000 P beeffe7071f041e297918aa03aa556c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beeffe7071f041e297918aa03aa556c9" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 35469 } }
I20260925 10:39:14.539866 25754 raft_consensus.cc:388] T 00000000000000000000000000000000 P beeffe7071f041e297918aa03aa556c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:39:14.540009 25754 raft_consensus.cc:482] T 00000000000000000000000000000000 P beeffe7071f041e297918aa03aa556c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:39:14.540222 25754 raft_consensus.cc:3037] T 00000000000000000000000000000000 P beeffe7071f041e297918aa03aa556c9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:39:14.542241 25754 raft_consensus.cc:504] T 00000000000000000000000000000000 P beeffe7071f041e297918aa03aa556c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beeffe7071f041e297918aa03aa556c9" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 35469 } }
I20260925 10:39:14.542768 25754 leader_election.cc:302] T 00000000000000000000000000000000 P beeffe7071f041e297918aa03aa556c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: beeffe7071f041e297918aa03aa556c9; no voters:
I20260925 10:39:14.543286 25754 leader_election.cc:288] T 00000000000000000000000000000000 P beeffe7071f041e297918aa03aa556c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:39:14.543448 25759 raft_consensus.cc:2781] T 00000000000000000000000000000000 P beeffe7071f041e297918aa03aa556c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:39:14.544198 25759 raft_consensus.cc:686] T 00000000000000000000000000000000 P beeffe7071f041e297918aa03aa556c9 [term 1 LEADER]: Becoming Leader. State: Replica: beeffe7071f041e297918aa03aa556c9, State: Running, Role: LEADER
I20260925 10:39:14.544754 25759 consensus_queue.cc:237] T 00000000000000000000000000000000 P beeffe7071f041e297918aa03aa556c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beeffe7071f041e297918aa03aa556c9" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 35469 } }
I20260925 10:39:14.545235 25754 sys_catalog.cc:564] T 00000000000000000000000000000000 P beeffe7071f041e297918aa03aa556c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:39:14.547631 25761 sys_catalog.cc:455] T 00000000000000000000000000000000 P beeffe7071f041e297918aa03aa556c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader beeffe7071f041e297918aa03aa556c9. Latest consensus state: current_term: 1 leader_uuid: "beeffe7071f041e297918aa03aa556c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beeffe7071f041e297918aa03aa556c9" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 35469 } } }
I20260925 10:39:14.547642 25760 sys_catalog.cc:455] T 00000000000000000000000000000000 P beeffe7071f041e297918aa03aa556c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "beeffe7071f041e297918aa03aa556c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beeffe7071f041e297918aa03aa556c9" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 35469 } } }
I20260925 10:39:14.548080 25760 sys_catalog.cc:458] T 00000000000000000000000000000000 P beeffe7071f041e297918aa03aa556c9 [sys.catalog]: This master's current role is: LEADER
I20260925 10:39:14.548080 25761 sys_catalog.cc:458] T 00000000000000000000000000000000 P beeffe7071f041e297918aa03aa556c9 [sys.catalog]: This master's current role is: LEADER
I20260925 10:39:14.549889 25766 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:39:14.555184 25766 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:39:14.564086 25766 catalog_manager.cc:1347] Generated new cluster ID: e4a8df5f6aad4f0aa862095ca1089ca1
I20260925 10:39:14.564208 25766 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:39:14.575703 25766 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:39:14.576920 25766 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:39:14.585974 25766 catalog_manager.cc:5868] T 00000000000000000000000000000000 P beeffe7071f041e297918aa03aa556c9: Generated new TSK 0
I20260925 10:39:14.586829 25766 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab.
Entry for principal kudu/127.23.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab.
I20260925 10:39:14.637547 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.1:0
--local_ip_for_outbound_sockets=127.23.100.1
--webserver_interface=127.23.100.1
--webserver_port=0
--tserver_master_addrs=127.23.100.62:35469
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab
--principal=kudu/127.23.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:39451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:39:14.951768 25780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:39:14.952078 25780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:39:14.952183 25780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:39:14.959297 25780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:39:14.959394 25780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:39:14.959473 25780 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 10:39:14.959527 25780 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 10:39:14.959592 25780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.1
I20260925 10:39:14.969455 25780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:39451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab
--principal=kudu/127.23.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.100.62:35469
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.1
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:39:14.970726 25780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:39:14.971894 25780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:39:14.984210 25785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:14.984239 25786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:14.984272 25788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:14.984866 25780 server_base.cc:1034] running on GCE node
I20260925 10:39:14.985368 25780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:39:14.986220 25780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:39:14.987432 25780 hybrid_clock.cc:648] HybridClock initialized: now 1790332754987405 us; error 56 us; skew 500 ppm
I20260925 10:39:14.987756 25780 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:39:14 dist-test-slave-dxdd krb5kdc[25664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332754, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:39:14.990929 25780 init.cc:377] Logged in from keytab as kudu/127.23.100.1@KRBTEST.COM (short username kudu)
I20260925 10:39:14.992233 25780 webserver.cc:460] Webserver started at http://127.23.100.1:36017/ using document root <none> and password file <none>
I20260925 10:39:14.992835 25780 fs_manager.cc:359] Metadata directory not provided
I20260925 10:39:14.992933 25780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:39:14.993242 25780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:39:14.996070 25780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-0/data/instance:
uuid: "5ae87327c48c44938d940dd9995a4d7e"
format_stamp: "Formatted at 2026-09-25 10:39:14 on dist-test-slave-dxdd"
server_key: "567a642d0e633c7f8cf1c15bae8ce48b"
server_key_iv: "f3c8744604d5220b5cf16f2838f874a6"
server_key_version: "encryptionkey@0"
I20260925 10:39:14.996745 25780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-0/wal/instance:
uuid: "5ae87327c48c44938d940dd9995a4d7e"
format_stamp: "Formatted at 2026-09-25 10:39:14 on dist-test-slave-dxdd"
server_key: "567a642d0e633c7f8cf1c15bae8ce48b"
server_key_iv: "f3c8744604d5220b5cf16f2838f874a6"
server_key_version: "encryptionkey@0"
I20260925 10:39:15.001535 25780 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 10:39:15.004751 25795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:15.006000 25780 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 10:39:15.006126 25780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-0/wal
uuid: "5ae87327c48c44938d940dd9995a4d7e"
format_stamp: "Formatted at 2026-09-25 10:39:14 on dist-test-slave-dxdd"
server_key: "567a642d0e633c7f8cf1c15bae8ce48b"
server_key_iv: "f3c8744604d5220b5cf16f2838f874a6"
server_key_version: "encryptionkey@0"
I20260925 10:39:15.006275 25780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:39:15.018626 25780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:39:15.022476 25780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:39:15.022714 25780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:39:15.025319 25780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:39:15.025444 25780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:15.025578 25780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:39:15.025650 25780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 10:39:15 dist-test-slave-dxdd krb5kdc[25664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332754, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.1@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:39:15.048877 25757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:15.025746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:39527 (local address 127.23.100.62:35469)
0925 10:39:15.026018 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:15.026028 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:15.027449 (+ 1421us) server_negotiation.cc:407] Connection header received
0925 10:39:15.030475 (+ 3026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:15.030489 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:15.030572 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:15.030681 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:15.032934 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:15.034247 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:15.036908 (+ 2661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:15.037598 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:15.037766 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:15.042334 (+ 4568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:15.042343 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:15.042349 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:15.042393 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:15.045822 (+ 3429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:15.046606 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:15.046611 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:15.046614 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:15.046755 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:15.047180 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:15.047185 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:15.047188 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:15.047437 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:15.047585 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:15.048261 (+ 676us) server_negotiation.cc:299] Negotiation successful
0925 10:39:15.048506 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 10:39:15.049876 25804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:15.026397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35469 (local address 127.23.100.1:39527)
0925 10:39:15.027256 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:15.027307 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:15.030110 (+ 2803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:15.031065 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:15.031082 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:15.031859 (+ 777us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:15.032682 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:15.032731 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:15.034442 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:15.034449 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:15.036763 (+ 2314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:15.036775 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:15.038057 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:15.038062 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:15.038304 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:15.039121 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:15.039142 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:15.042057 (+ 2915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:15.045946 (+ 3889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:15.045955 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:15.045994 (+ 39us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:15.046470 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:15.046868 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:15.046874 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:15.046877 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:15.047067 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:15.047563 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:15.047570 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:15.047918 (+ 348us) client_negotiation.cc:769] Sending connection context
0925 10:39:15.048166 (+ 248us) client_negotiation.cc:240] Negotiation successful
0925 10:39:15.048648 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":334,"threads_started":1}
I20260925 10:39:15.075639 25757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:15.060895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:55901 (local address 127.23.100.62:35469)
0925 10:39:15.061056 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:15.061060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:15.061697 (+ 637us) server_negotiation.cc:407] Connection header received
0925 10:39:15.061895 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:15.061900 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:15.061955 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:15.062058 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:15.063364 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:15.064385 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:15.066676 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:15.067277 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:15.067395 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:15.069566 (+ 2171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:15.069572 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:15.069576 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:15.069613 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:15.072650 (+ 3037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:15.073534 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:15.073541 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:15.073546 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:15.073695 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:15.074172 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:15.074178 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:15.074183 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:15.074510 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:15.074656 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:15.075071 (+ 415us) server_negotiation.cc:299] Negotiation successful
0925 10:39:15.075284 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 10:39:15.075810 25858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:15.060971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35469 (local address 127.23.100.1:55901)
0925 10:39:15.061601 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:15.061615 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:15.061761 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:15.062081 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:15.062086 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:15.062706 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:15.063233 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:15.063243 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:15.064539 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:15.064548 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:15.066529 (+ 1981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:15.066540 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:15.067682 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:15.067691 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:15.068002 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:15.068702 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:15.068719 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:15.069387 (+ 668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:15.072770 (+ 3383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:15.072775 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:15.072778 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:15.073344 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:15.073830 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:15.073835 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:15.073839 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:15.074010 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:15.074638 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:15.074643 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:15.074822 (+ 179us) client_negotiation.cc:769] Sending connection context
0925 10:39:15.074975 (+ 153us) client_negotiation.cc:240] Negotiation successful
0925 10:39:15.075304 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":370,"threads_started":1}
I20260925 10:39:15.086180 25780 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.1:42361
I20260925 10:39:15.086181 25914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.1:42361 every 8 connection(s)
I20260925 10:39:15.087428 25780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
I20260925 10:39:15.089344 25915 heartbeater.cc:347] Connected to a master server at 127.23.100.62:35469
I20260925 10:39:15.089564 25915 heartbeater.cc:464] Registering TS with master...
I20260925 10:39:15.090175 25915 heartbeater.cc:511] Master 127.23.100.62:35469 requested a full tablet report, sending...
I20260925 10:39:15.092007 25709 ts_manager.cc:194] Registered new tserver with Master: 5ae87327c48c44938d940dd9995a4d7e (127.23.100.1:42361)
I20260925 10:39:15.093407 25709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.1@KRBTEST.COM'} at 127.23.100.1:39527
I20260925 10:39:15.094908 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 25780
I20260925 10:39:15.095113 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-0/wal/instance
I20260925 10:39:15.095700 23952 external_mini_cluster.cc:1428] Setting key 7c504e0724491655a6dbeb7184a6cea1
WARNING: no policy specified for kudu/127.23.100.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab.
Entry for principal kudu/127.23.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab.
I20260925 10:39:15.135102 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.2:0
--local_ip_for_outbound_sockets=127.23.100.2
--webserver_interface=127.23.100.2
--webserver_port=0
--tserver_master_addrs=127.23.100.62:35469
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab
--principal=kudu/127.23.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:39451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:39:15.445446 25919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:39:15.445757 25919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:39:15.445938 25919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:39:15.453264 25919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:39:15.453377 25919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:39:15.453522 25919 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 10:39:15.453598 25919 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 10:39:15.453716 25919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.2
I20260925 10:39:15.463421 25919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:39451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab
--principal=kudu/127.23.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.100.62:35469
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.2
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:39:15.464654 25919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:39:15.465971 25919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:39:15.479000 25924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:15.479079 25925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:15.479044 25927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:15.479836 25919 server_base.cc:1034] running on GCE node
I20260925 10:39:15.480438 25919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:39:15.481324 25919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:39:15.482539 25919 hybrid_clock.cc:648] HybridClock initialized: now 1790332755482488 us; error 61 us; skew 500 ppm
I20260925 10:39:15.482842 25919 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:39:15 dist-test-slave-dxdd krb5kdc[25664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332755, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:39:15.485728 25919 init.cc:377] Logged in from keytab as kudu/127.23.100.2@KRBTEST.COM (short username kudu)
I20260925 10:39:15.487150 25919 webserver.cc:460] Webserver started at http://127.23.100.2:41215/ using document root <none> and password file <none>
I20260925 10:39:15.487753 25919 fs_manager.cc:359] Metadata directory not provided
I20260925 10:39:15.487849 25919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:39:15.488103 25919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:39:15.490895 25919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-1/data/instance:
uuid: "b87295069e6d49a49c4e68fb5af6fce0"
format_stamp: "Formatted at 2026-09-25 10:39:15 on dist-test-slave-dxdd"
server_key: "909cbe034827885e2bfb805b9836dc38"
server_key_iv: "f351fecdb2835b3d2d812ecdf6c4fd0f"
server_key_version: "encryptionkey@0"
I20260925 10:39:15.491571 25919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-1/wal/instance:
uuid: "b87295069e6d49a49c4e68fb5af6fce0"
format_stamp: "Formatted at 2026-09-25 10:39:15 on dist-test-slave-dxdd"
server_key: "909cbe034827885e2bfb805b9836dc38"
server_key_iv: "f351fecdb2835b3d2d812ecdf6c4fd0f"
server_key_version: "encryptionkey@0"
I20260925 10:39:15.496203 25919 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 10:39:15.499367 25934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:15.500645 25919 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:39:15.500800 25919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-1/wal
uuid: "b87295069e6d49a49c4e68fb5af6fce0"
format_stamp: "Formatted at 2026-09-25 10:39:15 on dist-test-slave-dxdd"
server_key: "909cbe034827885e2bfb805b9836dc38"
server_key_iv: "f351fecdb2835b3d2d812ecdf6c4fd0f"
server_key_version: "encryptionkey@0"
I20260925 10:39:15.501009 25919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:39:15.516117 25919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:39:15.519788 25919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:39:15.520074 25919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:39:15.522465 25919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:39:15.522583 25919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:15.522694 25919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:39:15.522778 25919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 10:39:15 dist-test-slave-dxdd krb5kdc[25664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332755, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.2@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:39:15.546242 25757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:15.523045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:48947 (local address 127.23.100.62:35469)
0925 10:39:15.523242 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:15.523250 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:15.524568 (+ 1318us) server_negotiation.cc:407] Connection header received
0925 10:39:15.526612 (+ 2044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:15.526618 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:15.526707 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:15.526814 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:15.528856 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:15.529838 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:15.533123 (+ 3285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:15.533667 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:15.533802 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:15.538536 (+ 4734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:15.538547 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:15.538554 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:15.538608 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:15.542714 (+ 4106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:15.543711 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:15.543717 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:15.543722 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:15.543894 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:15.544508 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:15.544515 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:15.544518 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:15.544866 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:15.545015 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:15.545638 (+ 623us) server_negotiation.cc:299] Negotiation successful
0925 10:39:15.545853 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 10:39:15.547232 25943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:15.523681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35469 (local address 127.23.100.2:48947)
0925 10:39:15.524425 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:15.524461 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:15.526232 (+ 1771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:15.527067 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:15.527080 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:15.527830 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:15.528682 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:15.528702 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:15.530021 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:15.530030 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:15.532988 (+ 2958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:15.533002 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:15.533895 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:15.533902 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:15.534177 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:15.534940 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:15.534976 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:15.538249 (+ 3273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:15.542872 (+ 4623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:15.542886 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:15.542905 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:15.543585 (+ 680us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:15.544021 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:15.544029 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:15.544038 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:15.544357 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:15.544983 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:15.544991 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:15.545314 (+ 323us) client_negotiation.cc:769] Sending connection context
0925 10:39:15.545550 (+ 236us) client_negotiation.cc:240] Negotiation successful
0925 10:39:15.546087 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":335,"threads_started":1}
I20260925 10:39:15.569979 26011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:15.555165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35469 (local address 127.23.100.2:35331)
0925 10:39:15.555965 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:15.555981 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:15.556139 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:15.556434 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:15.556438 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:15.556859 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:15.557126 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:15.557134 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:15.558124 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:15.558131 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:15.559692 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:15.559709 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:15.561009 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:15.561013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:15.561197 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:15.561944 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:15.561991 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:15.563259 (+ 1268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:15.566905 (+ 3646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:15.566917 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:15.566924 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:15.567670 (+ 746us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:15.568182 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:15.568189 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:15.568193 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:15.568367 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:15.569141 (+ 774us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:15.569149 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:15.569340 (+ 191us) client_negotiation.cc:769] Sending connection context
0925 10:39:15.569449 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 10:39:15.569665 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":502,"threads_started":1}
I20260925 10:39:15.570438 25757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:15.555237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:35331 (local address 127.23.100.62:35469)
0925 10:39:15.555478 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:15.555486 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:15.556075 (+ 589us) server_negotiation.cc:407] Connection header received
0925 10:39:15.556249 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:15.556257 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:15.556317 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:15.556412 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:15.557255 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:15.557993 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:15.559867 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:15.560646 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:15.560783 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:15.563537 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:15.563546 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:15.563549 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:15.563577 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:15.566773 (+ 3196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:15.567855 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:15.567862 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:15.567867 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:15.568058 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:15.568507 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:15.568513 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:15.568516 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:15.568989 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:15.569172 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:15.569608 (+ 436us) server_negotiation.cc:299] Negotiation successful
0925 10:39:15.569901 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 10:39:15.575135 25919 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.2:33655
I20260925 10:39:15.575166 26053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.2:33655 every 8 connection(s)
I20260925 10:39:15.576252 25919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
I20260925 10:39:15.578110 26054 heartbeater.cc:347] Connected to a master server at 127.23.100.62:35469
I20260925 10:39:15.578361 26054 heartbeater.cc:464] Registering TS with master...
I20260925 10:39:15.578964 26054 heartbeater.cc:511] Master 127.23.100.62:35469 requested a full tablet report, sending...
I20260925 10:39:15.580230 25709 ts_manager.cc:194] Registered new tserver with Master: b87295069e6d49a49c4e68fb5af6fce0 (127.23.100.2:33655)
I20260925 10:39:15.581007 25709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.2@KRBTEST.COM'} at 127.23.100.2:48947
I20260925 10:39:15.581748 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 25919
I20260925 10:39:15.581934 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-1/wal/instance
I20260925 10:39:15.582433 23952 external_mini_cluster.cc:1428] Setting key bab69429620da27401d1aa71b21cf612
WARNING: no policy specified for kudu/127.23.100.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab.
Entry for principal kudu/127.23.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab.
I20260925 10:39:15.621490 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.3:0
--local_ip_for_outbound_sockets=127.23.100.3
--webserver_interface=127.23.100.3
--webserver_port=0
--tserver_master_addrs=127.23.100.62:35469
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab
--principal=kudu/127.23.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:39451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:39:15.930779 26058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:39:15.931041 26058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:39:15.931135 26058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:39:15.938149 26058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:39:15.938248 26058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:39:15.938346 26058 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 10:39:15.938401 26058 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 10:39:15.938462 26058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.3
I20260925 10:39:15.948048 26058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:39451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab
--principal=kudu/127.23.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.100.62:35469
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.3
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:39:15.949208 26058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:39:15.950443 26058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:39:15.963672 26063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:15.963671 26066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:15.963717 26064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:15.964414 26058 server_base.cc:1034] running on GCE node
I20260925 10:39:15.964923 26058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:39:15.965858 26058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:39:15.967067 26058 hybrid_clock.cc:648] HybridClock initialized: now 1790332755967024 us; error 32 us; skew 500 ppm
I20260925 10:39:15.967375 26058 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:39:15 dist-test-slave-dxdd krb5kdc[25664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332755, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:39:15.974381 26058 init.cc:377] Logged in from keytab as kudu/127.23.100.3@KRBTEST.COM (short username kudu)
I20260925 10:39:15.975895 26058 webserver.cc:460] Webserver started at http://127.23.100.3:35349/ using document root <none> and password file <none>
I20260925 10:39:15.976650 26058 fs_manager.cc:359] Metadata directory not provided
I20260925 10:39:15.976787 26058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:39:15.977103 26058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:39:15.980090 26058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-2/data/instance:
uuid: "f216ed6b172f40999704a36c3b47bfdd"
format_stamp: "Formatted at 2026-09-25 10:39:15 on dist-test-slave-dxdd"
server_key: "54f081fae7fc684caf3d1386a3820bc1"
server_key_iv: "60117a09b0cc68063eaed7b8c2f65635"
server_key_version: "encryptionkey@0"
I20260925 10:39:15.980846 26058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-2/wal/instance:
uuid: "f216ed6b172f40999704a36c3b47bfdd"
format_stamp: "Formatted at 2026-09-25 10:39:15 on dist-test-slave-dxdd"
server_key: "54f081fae7fc684caf3d1386a3820bc1"
server_key_iv: "60117a09b0cc68063eaed7b8c2f65635"
server_key_version: "encryptionkey@0"
I20260925 10:39:15.986397 26058 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260925 10:39:15.990010 26073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:15.991508 26058 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 10:39:15.991681 26058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-2/wal
uuid: "f216ed6b172f40999704a36c3b47bfdd"
format_stamp: "Formatted at 2026-09-25 10:39:15 on dist-test-slave-dxdd"
server_key: "54f081fae7fc684caf3d1386a3820bc1"
server_key_iv: "60117a09b0cc68063eaed7b8c2f65635"
server_key_version: "encryptionkey@0"
I20260925 10:39:15.991900 26058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:39:16.003854 26058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:39:16.007994 26058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:39:16.008279 26058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:39:16.011072 26058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:39:16.011195 26058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:16.011332 26058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:39:16.011427 26058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 10:39:16 dist-test-slave-dxdd krb5kdc[25664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332755, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.3@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:39:16.034328 25757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:16.011611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:35151 (local address 127.23.100.62:35469)
0925 10:39:16.011778 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:16.011783 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:16.013386 (+ 1603us) server_negotiation.cc:407] Connection header received
0925 10:39:16.015436 (+ 2050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:16.015442 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:16.015539 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:16.015650 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:16.017735 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:16.019077 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:16.021609 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:16.022035 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:16.022173 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:16.027232 (+ 5059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:16.027241 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:16.027249 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:16.027298 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:16.030895 (+ 3597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:16.031827 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:16.031833 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:16.031837 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:16.032044 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:16.032586 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:16.032593 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:16.032596 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:16.032866 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:16.033027 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:16.033733 (+ 706us) server_negotiation.cc:299] Negotiation successful
0925 10:39:16.033958 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 10:39:16.035470 26082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:16.012430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35469 (local address 127.23.100.3:35151)
0925 10:39:16.013234 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:16.013270 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:16.015156 (+ 1886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:16.015884 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:16.015892 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:16.016664 (+ 772us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:16.017502 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:16.017547 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:16.019264 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:16.019269 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:16.021440 (+ 2171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:16.021457 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:16.022282 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:16.022290 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:16.022535 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:16.023449 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:16.023491 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:16.026910 (+ 3419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:16.031033 (+ 4123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:16.031042 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:16.031061 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:16.031681 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:16.032183 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:16.032194 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:16.032197 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:16.032456 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:16.033010 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:16.033018 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:16.033369 (+ 351us) client_negotiation.cc:769] Sending connection context
0925 10:39:16.033634 (+ 265us) client_negotiation.cc:240] Negotiation successful
0925 10:39:16.034161 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":362,"threads_started":1}
I20260925 10:39:16.064383 26058 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.3:41541
I20260925 10:39:16.064394 26192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.3:41541 every 8 connection(s)
I20260925 10:39:16.065500 26150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:16.048236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35469 (local address 127.23.100.3:53007)
0925 10:39:16.049180 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:16.049203 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:16.049362 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:16.049762 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:16.049768 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:16.050580 (+ 812us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:16.051082 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:16.051102 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:16.052810 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:16.052817 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:16.054955 (+ 2138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:16.054975 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:16.056106 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:16.056116 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:16.056344 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:16.057184 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:16.057217 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:16.058258 (+ 1041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:16.062707 (+ 4449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:16.062711 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:16.062714 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:16.063091 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:16.063644 (+ 553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:16.063649 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:16.063653 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:16.063853 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:16.064662 (+ 809us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:16.064667 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:16.064837 (+ 170us) client_negotiation.cc:769] Sending connection context
0925 10:39:16.064946 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 10:39:16.065191 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"thread_start_us":512,"threads_started":1}
I20260925 10:39:16.065712 25757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:16.047982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:53007 (local address 127.23.100.62:35469)
0925 10:39:16.048179 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:16.048187 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:16.049324 (+ 1137us) server_negotiation.cc:407] Connection header received
0925 10:39:16.049485 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:16.049494 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:16.049599 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:16.049714 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:16.051287 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:16.052657 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:16.055122 (+ 2465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:16.055859 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:16.056012 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:16.058512 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:16.058521 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:16.058526 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:16.058567 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:16.062590 (+ 4023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:16.063276 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:16.063284 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:16.063290 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:16.063471 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:16.064031 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:16.064036 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:16.064039 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:16.064460 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:16.064654 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:16.065106 (+ 452us) server_negotiation.cc:299] Negotiation successful
0925 10:39:16.065319 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 10:39:16.066128 26058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
I20260925 10:39:16.068245 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 26058
I20260925 10:39:16.068425 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-2/wal/instance
I20260925 10:39:16.068811 23952 external_mini_cluster.cc:1428] Setting key 7edaabd0cdd64266851739ac89a821eb
I20260925 10:39:16.070029 26193 heartbeater.cc:347] Connected to a master server at 127.23.100.62:35469
I20260925 10:39:16.070334 26193 heartbeater.cc:464] Registering TS with master...
I20260925 10:39:16.071045 26193 heartbeater.cc:511] Master 127.23.100.62:35469 requested a full tablet report, sending...
I20260925 10:39:16.072453 25709 ts_manager.cc:194] Registered new tserver with Master: f216ed6b172f40999704a36c3b47bfdd (127.23.100.3:41541)
I20260925 10:39:16.073242 25709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.3@KRBTEST.COM'} at 127.23.100.3:35151
I20260925 10:39:16.084415 23952 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:39:16.095950 25915 heartbeater.cc:503] Master 127.23.100.62:35469 was elected leader, sending a full tablet report...
Sep 25 10:39:16 dist-test-slave-dxdd krb5kdc[25664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 10:39:16 dist-test-slave-dxdd krb5kdc[25664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:39:16.130200 25757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:16.116868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43768 (local address 127.23.100.62:35469)
0925 10:39:16.117132 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:16.117139 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:16.117427 (+ 288us) server_negotiation.cc:407] Connection header received
0925 10:39:16.117575 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:16.117580 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:16.117658 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:16.117744 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:16.119162 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:16.119882 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:16.121039 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:16.121418 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:16.121531 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:16.125273 (+ 3742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:16.125278 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:16.125281 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:16.125310 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:16.127720 (+ 2410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:16.128345 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:16.128358 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:16.128361 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:16.128490 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:16.128859 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:16.128864 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:16.128867 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:16.129122 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:16.129380 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:16.129682 (+ 302us) server_negotiation.cc:299] Negotiation successful
0925 10:39:16.129896 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260925 10:39:16.134176 25709 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43768:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:39:16.141166 25709 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:39:16.164248 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:16.155590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53270 (local address 127.23.100.2:33655)
0925 10:39:16.156177 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:16.156189 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:16.156221 (+ 32us) server_negotiation.cc:407] Connection header received
0925 10:39:16.156339 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:16.156349 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:16.156633 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:16.156864 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:39:16.157435 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:16.159306 (+ 1871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:16.161570 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:16.162960 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:16.163096 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:16.163255 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:16.163828 (+ 573us) server_negotiation.cc:299] Negotiation successful
0925 10:39:16.163985 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":309,"threads_started":1}
I20260925 10:39:16.164288 26210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:16.155204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.2:33655 (local address 127.0.0.1:53270)
0925 10:39:16.155748 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:16.155768 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:16.155890 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:16.156876 (+ 986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:16.156881 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:16.156906 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:39:16.157239 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:16.157255 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:16.159537 (+ 2282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:16.159546 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:16.161342 (+ 1796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:16.161356 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:16.163377 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:16.163381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:16.163535 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:16.163568 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 10:39:16.163741 (+ 173us) client_negotiation.cc:240] Negotiation successful
0925 10:39:16.163928 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":554,"threads_started":1}
I20260925 10:39:16.167476 26206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:16.153183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:42361 (local address 127.0.0.1:33618)
0925 10:39:16.155151 (+ 1968us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:16.155171 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:16.155293 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:16.159351 (+ 4058us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:16.159355 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:16.159379 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:39:16.159739 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:16.159748 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:16.162583 (+ 2835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:16.162588 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:16.164842 (+ 2254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:16.164862 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:16.166774 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:16.166780 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:16.166924 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:16.166953 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 10:39:16.167026 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 10:39:16.167147 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1801,"thread_start_us":354,"threads_started":1}
I20260925 10:39:16.168075 26207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:16.153051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.3:41541 (local address 127.0.0.1:35528)
0925 10:39:16.154062 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:16.154110 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:16.154348 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:16.160614 (+ 6266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:16.160619 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:16.160641 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:39:16.161083 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:16.161096 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:16.163101 (+ 2005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:16.163111 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:16.165317 (+ 2206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:16.165331 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:16.167179 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:16.167188 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:16.167367 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:16.167398 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 10:39:16.167461 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 10:39:16.167724 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":1803,"threads_started":1}
I20260925 10:39:16.168411 26209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:16.153765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33618 (local address 127.23.100.1:42361)
0925 10:39:16.158753 (+ 4988us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:16.158761 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:16.158789 (+ 28us) server_negotiation.cc:407] Connection header received
0925 10:39:16.158881 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:16.158890 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:16.159142 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:16.160966 (+ 1824us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:39:16.161210 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:16.162425 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:16.165051 (+ 2626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:16.166533 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:16.167718 (+ 1185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:16.167902 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:16.167968 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 10:39:16.168149 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4832,"thread_start_us":4711,"threads_started":1}
I20260925 10:39:16.169060 26208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:16.153358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35528 (local address 127.23.100.3:41541)
0925 10:39:16.160212 (+ 6854us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:16.160217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:16.160240 (+ 23us) server_negotiation.cc:407] Connection header received
0925 10:39:16.160303 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:16.160309 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:16.160449 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:16.160629 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:39:16.161336 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:16.162959 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:16.165488 (+ 2529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:16.166958 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:16.168290 (+ 1332us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:16.168509 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:16.168573 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 10:39:16.168725 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6729,"thread_start_us":6579,"threads_started":1}
I20260925 10:39:16.168536 25984 tablet_service.cc:1467] Processing CreateTablet for tablet a6cfbc1f6f664e65bde7a073df41da39 (DEFAULT_TABLE table=test-table [id=cdc555a62b304e51b28550247bc669ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:39:16.170037 25984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6cfbc1f6f664e65bde7a073df41da39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:39:16.172466 25845 tablet_service.cc:1467] Processing CreateTablet for tablet a6cfbc1f6f664e65bde7a073df41da39 (DEFAULT_TABLE table=test-table [id=cdc555a62b304e51b28550247bc669ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:39:16.173277 26123 tablet_service.cc:1467] Processing CreateTablet for tablet a6cfbc1f6f664e65bde7a073df41da39 (DEFAULT_TABLE table=test-table [id=cdc555a62b304e51b28550247bc669ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:39:16.173771 25845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6cfbc1f6f664e65bde7a073df41da39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:39:16.174592 26123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6cfbc1f6f664e65bde7a073df41da39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:39:16.182843 26212 tablet_bootstrap.cc:492] T a6cfbc1f6f664e65bde7a073df41da39 P b87295069e6d49a49c4e68fb5af6fce0: Bootstrap starting.
I20260925 10:39:16.185675 26212 tablet_bootstrap.cc:654] T a6cfbc1f6f664e65bde7a073df41da39 P b87295069e6d49a49c4e68fb5af6fce0: Neither blocks nor log segments found. Creating new log.
I20260925 10:39:16.186739 26212 log.cc:824] T a6cfbc1f6f664e65bde7a073df41da39 P b87295069e6d49a49c4e68fb5af6fce0: Log is configured to *not* fsync() on all Append() calls
I20260925 10:39:16.188735 26215 tablet_bootstrap.cc:492] T a6cfbc1f6f664e65bde7a073df41da39 P f216ed6b172f40999704a36c3b47bfdd: Bootstrap starting.
I20260925 10:39:16.187968 26214 tablet_bootstrap.cc:492] T a6cfbc1f6f664e65bde7a073df41da39 P 5ae87327c48c44938d940dd9995a4d7e: Bootstrap starting.
I20260925 10:39:16.189914 26212 tablet_bootstrap.cc:492] T a6cfbc1f6f664e65bde7a073df41da39 P b87295069e6d49a49c4e68fb5af6fce0: No bootstrap required, opened a new log
I20260925 10:39:16.190196 26212 ts_tablet_manager.cc:1397] T a6cfbc1f6f664e65bde7a073df41da39 P b87295069e6d49a49c4e68fb5af6fce0: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260925 10:39:16.192397 26214 tablet_bootstrap.cc:654] T a6cfbc1f6f664e65bde7a073df41da39 P 5ae87327c48c44938d940dd9995a4d7e: Neither blocks nor log segments found. Creating new log.
I20260925 10:39:16.193616 26214 log.cc:824] T a6cfbc1f6f664e65bde7a073df41da39 P 5ae87327c48c44938d940dd9995a4d7e: Log is configured to *not* fsync() on all Append() calls
I20260925 10:39:16.196619 26214 tablet_bootstrap.cc:492] T a6cfbc1f6f664e65bde7a073df41da39 P 5ae87327c48c44938d940dd9995a4d7e: No bootstrap required, opened a new log
I20260925 10:39:16.196877 26214 ts_tablet_manager.cc:1397] T a6cfbc1f6f664e65bde7a073df41da39 P 5ae87327c48c44938d940dd9995a4d7e: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260925 10:39:16.196924 26215 tablet_bootstrap.cc:654] T a6cfbc1f6f664e65bde7a073df41da39 P f216ed6b172f40999704a36c3b47bfdd: Neither blocks nor log segments found. Creating new log.
I20260925 10:39:16.198604 26215 log.cc:824] T a6cfbc1f6f664e65bde7a073df41da39 P f216ed6b172f40999704a36c3b47bfdd: Log is configured to *not* fsync() on all Append() calls
I20260925 10:39:16.200515 26212 raft_consensus.cc:348] T a6cfbc1f6f664e65bde7a073df41da39 P b87295069e6d49a49c4e68fb5af6fce0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } } peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } }
I20260925 10:39:16.200945 26212 raft_consensus.cc:374] T a6cfbc1f6f664e65bde7a073df41da39 P b87295069e6d49a49c4e68fb5af6fce0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:39:16.201000 26215 tablet_bootstrap.cc:492] T a6cfbc1f6f664e65bde7a073df41da39 P f216ed6b172f40999704a36c3b47bfdd: No bootstrap required, opened a new log
I20260925 10:39:16.201059 26212 raft_consensus.cc:729] T a6cfbc1f6f664e65bde7a073df41da39 P b87295069e6d49a49c4e68fb5af6fce0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87295069e6d49a49c4e68fb5af6fce0, State: Initialized, Role: FOLLOWER
I20260925 10:39:16.201263 26215 ts_tablet_manager.cc:1397] T a6cfbc1f6f664e65bde7a073df41da39 P f216ed6b172f40999704a36c3b47bfdd: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20260925 10:39:16.201644 26212 consensus_queue.cc:260] T a6cfbc1f6f664e65bde7a073df41da39 P b87295069e6d49a49c4e68fb5af6fce0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } } peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } }
I20260925 10:39:16.202750 26054 heartbeater.cc:503] Master 127.23.100.62:35469 was elected leader, sending a full tablet report...
I20260925 10:39:16.203325 26212 ts_tablet_manager.cc:1428] T a6cfbc1f6f664e65bde7a073df41da39 P b87295069e6d49a49c4e68fb5af6fce0: Time spent starting tablet: real 0.013s user 0.013s sys 0.001s
I20260925 10:39:16.206779 26214 raft_consensus.cc:348] T a6cfbc1f6f664e65bde7a073df41da39 P 5ae87327c48c44938d940dd9995a4d7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } } peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } }
I20260925 10:39:16.207177 26214 raft_consensus.cc:374] T a6cfbc1f6f664e65bde7a073df41da39 P 5ae87327c48c44938d940dd9995a4d7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:39:16.207285 26214 raft_consensus.cc:729] T a6cfbc1f6f664e65bde7a073df41da39 P 5ae87327c48c44938d940dd9995a4d7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ae87327c48c44938d940dd9995a4d7e, State: Initialized, Role: FOLLOWER
I20260925 10:39:16.207894 26214 consensus_queue.cc:260] T a6cfbc1f6f664e65bde7a073df41da39 P 5ae87327c48c44938d940dd9995a4d7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } } peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } }
I20260925 10:39:16.210070 26215 raft_consensus.cc:348] T a6cfbc1f6f664e65bde7a073df41da39 P f216ed6b172f40999704a36c3b47bfdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } } peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } }
I20260925 10:39:16.210464 26215 raft_consensus.cc:374] T a6cfbc1f6f664e65bde7a073df41da39 P f216ed6b172f40999704a36c3b47bfdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:39:16.210587 26215 raft_consensus.cc:729] T a6cfbc1f6f664e65bde7a073df41da39 P f216ed6b172f40999704a36c3b47bfdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f216ed6b172f40999704a36c3b47bfdd, State: Initialized, Role: FOLLOWER
I20260925 10:39:16.211488 26214 ts_tablet_manager.cc:1428] T a6cfbc1f6f664e65bde7a073df41da39 P 5ae87327c48c44938d940dd9995a4d7e: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260925 10:39:16.212121 26215 consensus_queue.cc:260] T a6cfbc1f6f664e65bde7a073df41da39 P f216ed6b172f40999704a36c3b47bfdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } } peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } }
I20260925 10:39:16.213215 26193 heartbeater.cc:503] Master 127.23.100.62:35469 was elected leader, sending a full tablet report...
I20260925 10:39:16.213708 26215 ts_tablet_manager.cc:1428] T a6cfbc1f6f664e65bde7a073df41da39 P f216ed6b172f40999704a36c3b47bfdd: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
W20260925 10:39:16.318812 26194 tablet.cc:2369] T a6cfbc1f6f664e65bde7a073df41da39 P f216ed6b172f40999704a36c3b47bfdd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:39:16.328418 26055 tablet.cc:2369] T a6cfbc1f6f664e65bde7a073df41da39 P b87295069e6d49a49c4e68fb5af6fce0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:39:16.339905 25916 tablet.cc:2369] T a6cfbc1f6f664e65bde7a073df41da39 P 5ae87327c48c44938d940dd9995a4d7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:39:16.448705 26218 raft_consensus.cc:482] T a6cfbc1f6f664e65bde7a073df41da39 P b87295069e6d49a49c4e68fb5af6fce0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:39:16.449090 26218 raft_consensus.cc:504] T a6cfbc1f6f664e65bde7a073df41da39 P b87295069e6d49a49c4e68fb5af6fce0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } } peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } }
I20260925 10:39:16.450918 26218 leader_election.cc:288] T a6cfbc1f6f664e65bde7a073df41da39 P b87295069e6d49a49c4e68fb5af6fce0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f216ed6b172f40999704a36c3b47bfdd (127.23.100.3:41541), 5ae87327c48c44938d940dd9995a4d7e (127.23.100.1:42361)
I20260925 10:39:16.457945 26209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:16.451518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:59247 (local address 127.23.100.1:42361)
0925 10:39:16.451718 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:16.451724 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:16.452583 (+ 859us) server_negotiation.cc:407] Connection header received
0925 10:39:16.452770 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:16.452776 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:16.452860 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:16.452948 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:39:16.453575 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:16.454406 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:16.456209 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:16.457044 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:16.457224 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:16.457291 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:16.457615 (+ 324us) server_negotiation.cc:299] Negotiation successful
0925 10:39:16.457728 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 10:39:16.457962 26208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:16.451775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:33941 (local address 127.23.100.3:41541)
0925 10:39:16.451973 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:16.451978 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:16.452803 (+ 825us) server_negotiation.cc:407] Connection header received
0925 10:39:16.452968 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:16.452973 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:16.453047 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:16.453142 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:39:16.453821 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:16.454545 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:16.456245 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:16.457017 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:16.457173 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:16.457261 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:16.457585 (+ 324us) server_negotiation.cc:299] Negotiation successful
0925 10:39:16.457703 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 10:39:16.458350 26222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:16.451277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.3:41541 (local address 127.23.100.2:33941)
0925 10:39:16.452697 (+ 1420us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:16.452715 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:16.452834 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:16.453182 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:16.453185 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:16.453228 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:39:16.453680 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:16.453691 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:16.454678 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:16.454681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:16.456078 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:16.456089 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:16.457241 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:16.457247 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:16.457434 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:16.457457 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 10:39:16.457532 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 10:39:16.457669 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1247,"mutex_wait_us":17,"spinlock_wait_cycles":6528,"thread_start_us":1014,"threads_started":1}
I20260925 10:39:16.458297 26220 raft_consensus.cc:482] T a6cfbc1f6f664e65bde7a073df41da39 P f216ed6b172f40999704a36c3b47bfdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:39:16.458448 26221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:16.451183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:42361 (local address 127.23.100.2:59247)
0925 10:39:16.452464 (+ 1281us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:16.452483 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:16.452632 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:16.452971 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:16.452987 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:16.453009 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:39:16.453432 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:16.453442 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:16.454533 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:16.454537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:16.456068 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:16.456079 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:16.457287 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:16.457293 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:16.457447 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:16.457469 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 10:39:16.457577 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 10:39:16.457700 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1057,"mutex_wait_us":45,"thread_start_us":747,"threads_started":1}
I20260925 10:39:16.459030 26220 raft_consensus.cc:504] T a6cfbc1f6f664e65bde7a073df41da39 P f216ed6b172f40999704a36c3b47bfdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } } peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } }
I20260925 10:39:16.459347 26147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6cfbc1f6f664e65bde7a073df41da39" candidate_uuid: "b87295069e6d49a49c4e68fb5af6fce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f216ed6b172f40999704a36c3b47bfdd" is_pre_election: true
I20260925 10:39:16.459506 25870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6cfbc1f6f664e65bde7a073df41da39" candidate_uuid: "b87295069e6d49a49c4e68fb5af6fce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ae87327c48c44938d940dd9995a4d7e" is_pre_election: true
I20260925 10:39:16.459897 26147 raft_consensus.cc:2454] T a6cfbc1f6f664e65bde7a073df41da39 P f216ed6b172f40999704a36c3b47bfdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87295069e6d49a49c4e68fb5af6fce0 in term 0.
I20260925 10:39:16.460026 25870 raft_consensus.cc:2454] T a6cfbc1f6f664e65bde7a073df41da39 P 5ae87327c48c44938d940dd9995a4d7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87295069e6d49a49c4e68fb5af6fce0 in term 0.
I20260925 10:39:16.460590 25937 leader_election.cc:302] T a6cfbc1f6f664e65bde7a073df41da39 P b87295069e6d49a49c4e68fb5af6fce0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5ae87327c48c44938d940dd9995a4d7e, b87295069e6d49a49c4e68fb5af6fce0, f216ed6b172f40999704a36c3b47bfdd; no voters:
I20260925 10:39:16.460968 26218 raft_consensus.cc:2781] T a6cfbc1f6f664e65bde7a073df41da39 P b87295069e6d49a49c4e68fb5af6fce0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:39:16.461127 26218 raft_consensus.cc:482] T a6cfbc1f6f664e65bde7a073df41da39 P b87295069e6d49a49c4e68fb5af6fce0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:39:16.461222 26218 raft_consensus.cc:3037] T a6cfbc1f6f664e65bde7a073df41da39 P b87295069e6d49a49c4e68fb5af6fce0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:39:16.462329 26220 leader_election.cc:288] T a6cfbc1f6f664e65bde7a073df41da39 P f216ed6b172f40999704a36c3b47bfdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ae87327c48c44938d940dd9995a4d7e (127.23.100.1:42361), b87295069e6d49a49c4e68fb5af6fce0 (127.23.100.2:33655)
I20260925 10:39:16.464375 26218 raft_consensus.cc:504] T a6cfbc1f6f664e65bde7a073df41da39 P b87295069e6d49a49c4e68fb5af6fce0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } } peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } }
I20260925 10:39:16.465342 25870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6cfbc1f6f664e65bde7a073df41da39" candidate_uuid: "b87295069e6d49a49c4e68fb5af6fce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ae87327c48c44938d940dd9995a4d7e"
I20260925 10:39:16.465337 26147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6cfbc1f6f664e65bde7a073df41da39" candidate_uuid: "b87295069e6d49a49c4e68fb5af6fce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f216ed6b172f40999704a36c3b47bfdd"
I20260925 10:39:16.465555 26147 raft_consensus.cc:3037] T a6cfbc1f6f664e65bde7a073df41da39 P f216ed6b172f40999704a36c3b47bfdd [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:39:16.465535 25870 raft_consensus.cc:3037] T a6cfbc1f6f664e65bde7a073df41da39 P 5ae87327c48c44938d940dd9995a4d7e [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:39:16.468225 26218 leader_election.cc:288] T a6cfbc1f6f664e65bde7a073df41da39 P b87295069e6d49a49c4e68fb5af6fce0 [CANDIDATE]: Term 1 election: Requested vote from peers f216ed6b172f40999704a36c3b47bfdd (127.23.100.3:41541), 5ae87327c48c44938d940dd9995a4d7e (127.23.100.1:42361)
I20260925 10:39:16.468730 26082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:16.461191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:42361 (local address 127.23.100.3:42433)
0925 10:39:16.461388 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:16.461407 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:16.461535 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:16.462271 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:16.462278 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:16.462338 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:39:16.462538 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:16.462549 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:16.463932 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:16.463937 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:16.466001 (+ 2064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:16.466019 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:16.467910 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:16.467917 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:16.468147 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:16.468182 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 10:39:16.468275 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 10:39:16.468443 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":16}
I20260925 10:39:16.468796 26209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:16.461830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:42433 (local address 127.23.100.1:42361)
0925 10:39:16.461986 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:16.461993 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:16.462009 (+ 16us) server_negotiation.cc:407] Connection header received
0925 10:39:16.462062 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:16.462066 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:16.462130 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:16.462669 (+ 539us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:39:16.462800 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:16.463786 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:16.466264 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:16.467733 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:16.468010 (+ 277us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:16.468084 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:16.468424 (+ 340us) server_negotiation.cc:299] Negotiation successful
0925 10:39:16.468535 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 10:39:16.468894 25870 raft_consensus.cc:2454] T a6cfbc1f6f664e65bde7a073df41da39 P 5ae87327c48c44938d940dd9995a4d7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87295069e6d49a49c4e68fb5af6fce0 in term 1.
I20260925 10:39:16.469322 25938 leader_election.cc:302] T a6cfbc1f6f664e65bde7a073df41da39 P b87295069e6d49a49c4e68fb5af6fce0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ae87327c48c44938d940dd9995a4d7e, b87295069e6d49a49c4e68fb5af6fce0; no voters:
I20260925 10:39:16.469688 26218 raft_consensus.cc:2781] T a6cfbc1f6f664e65bde7a073df41da39 P b87295069e6d49a49c4e68fb5af6fce0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:39:16.469550 25869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6cfbc1f6f664e65bde7a073df41da39" candidate_uuid: "f216ed6b172f40999704a36c3b47bfdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ae87327c48c44938d940dd9995a4d7e" is_pre_election: true
I20260925 10:39:16.469731 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:16.461847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:57087 (local address 127.23.100.2:33655)
0925 10:39:16.462029 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:16.462036 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:16.462056 (+ 20us) server_negotiation.cc:407] Connection header received
0925 10:39:16.462263 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:16.462270 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:16.462386 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:16.462460 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:39:16.463122 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:16.463822 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:16.466298 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:16.467510 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:16.467635 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:16.467710 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:16.469340 (+ 1630us) server_negotiation.cc:299] Negotiation successful
0925 10:39:16.469462 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 10:39:16.469729 26223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:16.461389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.2:33655 (local address 127.23.100.3:57087)
0925 10:39:16.461957 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:16.461986 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:16.462095 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:16.462652 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:16.462655 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:16.462673 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:39:16.462971 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:16.462993 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:16.464007 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:16.464011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:16.466137 (+ 2126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:16.466152 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:16.467910 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:16.467917 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:16.469176 (+ 1259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:16.469199 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 10:39:16.469301 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 10:39:16.469418 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":667,"threads_started":1}
I20260925 10:39:16.469954 25869 raft_consensus.cc:2379] T a6cfbc1f6f664e65bde7a073df41da39 P 5ae87327c48c44938d940dd9995a4d7e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f216ed6b172f40999704a36c3b47bfdd in current term 1: Already voted for candidate b87295069e6d49a49c4e68fb5af6fce0 in this term.
I20260925 10:39:16.468819 26147 raft_consensus.cc:2454] T a6cfbc1f6f664e65bde7a073df41da39 P f216ed6b172f40999704a36c3b47bfdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87295069e6d49a49c4e68fb5af6fce0 in term 1.
I20260925 10:39:16.470741 26218 raft_consensus.cc:686] T a6cfbc1f6f664e65bde7a073df41da39 P b87295069e6d49a49c4e68fb5af6fce0 [term 1 LEADER]: Becoming Leader. State: Replica: b87295069e6d49a49c4e68fb5af6fce0, State: Running, Role: LEADER
I20260925 10:39:16.470655 26006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6cfbc1f6f664e65bde7a073df41da39" candidate_uuid: "f216ed6b172f40999704a36c3b47bfdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87295069e6d49a49c4e68fb5af6fce0" is_pre_election: true
I20260925 10:39:16.471289 26218 consensus_queue.cc:237] T a6cfbc1f6f664e65bde7a073df41da39 P b87295069e6d49a49c4e68fb5af6fce0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } } peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } }
I20260925 10:39:16.471756 26076 leader_election.cc:302] T a6cfbc1f6f664e65bde7a073df41da39 P f216ed6b172f40999704a36c3b47bfdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f216ed6b172f40999704a36c3b47bfdd; no voters: 5ae87327c48c44938d940dd9995a4d7e, b87295069e6d49a49c4e68fb5af6fce0
I20260925 10:39:16.472195 26220 raft_consensus.cc:2726] T a6cfbc1f6f664e65bde7a073df41da39 P f216ed6b172f40999704a36c3b47bfdd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 10:39:16.475858 25709 catalog_manager.cc:5495] T a6cfbc1f6f664e65bde7a073df41da39 P b87295069e6d49a49c4e68fb5af6fce0 reported cstate change: term changed from 0 to 1, leader changed from <none> to b87295069e6d49a49c4e68fb5af6fce0 (127.23.100.2). New cstate: current_term: 1 leader_uuid: "b87295069e6d49a49c4e68fb5af6fce0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } health_report { overall_health: HEALTHY } } }
Sep 25 10:39:16 dist-test-slave-dxdd krb5kdc[25664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332754, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.62@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:39:16.561717 26231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:16.548591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35469 (local address 127.0.0.1:43778)
0925 10:39:16.549170 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:16.549205 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:16.549347 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:16.549631 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:16.549634 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:16.550407 (+ 773us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:16.550662 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:16.550670 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:16.551608 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:16.551612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:16.552625 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:16.552644 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:16.553301 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:16.553310 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:16.553437 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:16.553986 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:16.554012 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:16.556156 (+ 2144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:16.559182 (+ 3026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:16.559196 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:16.559203 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:16.559625 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:16.559995 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:16.559999 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:16.560002 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:16.560138 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:16.560596 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:16.560604 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:16.560793 (+ 189us) client_negotiation.cc:769] Sending connection context
0925 10:39:16.560966 (+ 173us) client_negotiation.cc:240] Negotiation successful
0925 10:39:16.561260 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":305,"threads_started":1}
I20260925 10:39:16.561738 25757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:16.548523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43778 (local address 127.23.100.62:35469)
0925 10:39:16.548678 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:16.548683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:16.549309 (+ 626us) server_negotiation.cc:407] Connection header received
0925 10:39:16.549443 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:16.549447 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:16.549520 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:16.549595 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:16.550773 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:16.551436 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:16.552742 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:16.553132 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:16.553220 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:16.556319 (+ 3099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:16.556324 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:16.556327 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:16.556353 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:16.559050 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:16.559749 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:16.559757 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:16.559761 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:16.559886 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:16.560234 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:16.560241 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:16.560244 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:16.560473 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:16.560622 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:16.561081 (+ 459us) server_negotiation.cc:299] Negotiation successful
0925 10:39:16.561380 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":119,"server-negotiator.queue_time_us":57}
I20260925 10:39:16.566259 25709 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:43778:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 10:39:16.567240 25709 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:39:16.573966 26123 tablet_service.cc:1467] Processing CreateTablet for tablet a997129f30954aaea55d2f8ce094a614 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=300552b0053244b291e8ece00b75b669]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 10:39:16.573971 25984 tablet_service.cc:1467] Processing CreateTablet for tablet a997129f30954aaea55d2f8ce094a614 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=300552b0053244b291e8ece00b75b669]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 10:39:16.575086 26123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a997129f30954aaea55d2f8ce094a614. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:39:16.575134 25984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a997129f30954aaea55d2f8ce094a614. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:39:16.579638 25845 tablet_service.cc:1467] Processing CreateTablet for tablet a997129f30954aaea55d2f8ce094a614 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=300552b0053244b291e8ece00b75b669]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 10:39:16.580524 25845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a997129f30954aaea55d2f8ce094a614. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:39:16.582453 26215 tablet_bootstrap.cc:492] T a997129f30954aaea55d2f8ce094a614 P f216ed6b172f40999704a36c3b47bfdd: Bootstrap starting.
I20260925 10:39:16.582551 26212 tablet_bootstrap.cc:492] T a997129f30954aaea55d2f8ce094a614 P b87295069e6d49a49c4e68fb5af6fce0: Bootstrap starting.
I20260925 10:39:16.584156 26212 tablet_bootstrap.cc:654] T a997129f30954aaea55d2f8ce094a614 P b87295069e6d49a49c4e68fb5af6fce0: Neither blocks nor log segments found. Creating new log.
I20260925 10:39:16.584156 26215 tablet_bootstrap.cc:654] T a997129f30954aaea55d2f8ce094a614 P f216ed6b172f40999704a36c3b47bfdd: Neither blocks nor log segments found. Creating new log.
I20260925 10:39:16.587962 26214 tablet_bootstrap.cc:492] T a997129f30954aaea55d2f8ce094a614 P 5ae87327c48c44938d940dd9995a4d7e: Bootstrap starting.
I20260925 10:39:16.589471 26214 tablet_bootstrap.cc:654] T a997129f30954aaea55d2f8ce094a614 P 5ae87327c48c44938d940dd9995a4d7e: Neither blocks nor log segments found. Creating new log.
I20260925 10:39:16.591763 26215 tablet_bootstrap.cc:492] T a997129f30954aaea55d2f8ce094a614 P f216ed6b172f40999704a36c3b47bfdd: No bootstrap required, opened a new log
I20260925 10:39:16.591915 26215 ts_tablet_manager.cc:1397] T a997129f30954aaea55d2f8ce094a614 P f216ed6b172f40999704a36c3b47bfdd: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260925 10:39:16.591955 26212 tablet_bootstrap.cc:492] T a997129f30954aaea55d2f8ce094a614 P b87295069e6d49a49c4e68fb5af6fce0: No bootstrap required, opened a new log
I20260925 10:39:16.592088 26212 ts_tablet_manager.cc:1397] T a997129f30954aaea55d2f8ce094a614 P b87295069e6d49a49c4e68fb5af6fce0: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260925 10:39:16.592193 26214 tablet_bootstrap.cc:492] T a997129f30954aaea55d2f8ce094a614 P 5ae87327c48c44938d940dd9995a4d7e: No bootstrap required, opened a new log
I20260925 10:39:16.592334 26214 ts_tablet_manager.cc:1397] T a997129f30954aaea55d2f8ce094a614 P 5ae87327c48c44938d940dd9995a4d7e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260925 10:39:16.592682 26215 raft_consensus.cc:348] T a997129f30954aaea55d2f8ce094a614 P f216ed6b172f40999704a36c3b47bfdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } } peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } }
I20260925 10:39:16.592964 26215 raft_consensus.cc:374] T a997129f30954aaea55d2f8ce094a614 P f216ed6b172f40999704a36c3b47bfdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:39:16.592876 26212 raft_consensus.cc:348] T a997129f30954aaea55d2f8ce094a614 P b87295069e6d49a49c4e68fb5af6fce0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } } peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } }
I20260925 10:39:16.593098 26215 raft_consensus.cc:729] T a997129f30954aaea55d2f8ce094a614 P f216ed6b172f40999704a36c3b47bfdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f216ed6b172f40999704a36c3b47bfdd, State: Initialized, Role: FOLLOWER
I20260925 10:39:16.593140 26212 raft_consensus.cc:374] T a997129f30954aaea55d2f8ce094a614 P b87295069e6d49a49c4e68fb5af6fce0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:39:16.593236 26212 raft_consensus.cc:729] T a997129f30954aaea55d2f8ce094a614 P b87295069e6d49a49c4e68fb5af6fce0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87295069e6d49a49c4e68fb5af6fce0, State: Initialized, Role: FOLLOWER
I20260925 10:39:16.593097 26214 raft_consensus.cc:348] T a997129f30954aaea55d2f8ce094a614 P 5ae87327c48c44938d940dd9995a4d7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } } peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } }
I20260925 10:39:16.593417 26214 raft_consensus.cc:374] T a997129f30954aaea55d2f8ce094a614 P 5ae87327c48c44938d940dd9995a4d7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:39:16.593508 26214 raft_consensus.cc:729] T a997129f30954aaea55d2f8ce094a614 P 5ae87327c48c44938d940dd9995a4d7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ae87327c48c44938d940dd9995a4d7e, State: Initialized, Role: FOLLOWER
I20260925 10:39:16.593420 26215 consensus_queue.cc:260] T a997129f30954aaea55d2f8ce094a614 P f216ed6b172f40999704a36c3b47bfdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } } peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } }
I20260925 10:39:16.593515 26212 consensus_queue.cc:260] T a997129f30954aaea55d2f8ce094a614 P b87295069e6d49a49c4e68fb5af6fce0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } } peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } }
I20260925 10:39:16.593796 26214 consensus_queue.cc:260] T a997129f30954aaea55d2f8ce094a614 P 5ae87327c48c44938d940dd9995a4d7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } } peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } }
I20260925 10:39:16.594015 26215 ts_tablet_manager.cc:1428] T a997129f30954aaea55d2f8ce094a614 P f216ed6b172f40999704a36c3b47bfdd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:39:16.593856 26220 tablet_replica.cc:439] T a997129f30954aaea55d2f8ce094a614 P f216ed6b172f40999704a36c3b47bfdd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } } peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } } }
I20260925 10:39:16.594110 26212 ts_tablet_manager.cc:1428] T a997129f30954aaea55d2f8ce094a614 P b87295069e6d49a49c4e68fb5af6fce0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:39:16.594132 26220 tablet_replica.cc:442] T a997129f30954aaea55d2f8ce094a614 P f216ed6b172f40999704a36c3b47bfdd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:39:16.594225 26219 tablet_replica.cc:439] T a997129f30954aaea55d2f8ce094a614 P 5ae87327c48c44938d940dd9995a4d7e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } } peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } } }
I20260925 10:39:16.594574 26219 tablet_replica.cc:442] T a997129f30954aaea55d2f8ce094a614 P 5ae87327c48c44938d940dd9995a4d7e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:39:16.594434 26218 tablet_replica.cc:439] T a997129f30954aaea55d2f8ce094a614 P b87295069e6d49a49c4e68fb5af6fce0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } } peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } } }
I20260925 10:39:16.594713 26218 tablet_replica.cc:442] T a997129f30954aaea55d2f8ce094a614 P b87295069e6d49a49c4e68fb5af6fce0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:39:16.596119 26214 ts_tablet_manager.cc:1428] T a997129f30954aaea55d2f8ce094a614 P 5ae87327c48c44938d940dd9995a4d7e: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260925 10:39:16.628022 26218 raft_consensus.cc:482] T a997129f30954aaea55d2f8ce094a614 P b87295069e6d49a49c4e68fb5af6fce0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:39:16.628242 26218 raft_consensus.cc:504] T a997129f30954aaea55d2f8ce094a614 P b87295069e6d49a49c4e68fb5af6fce0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } } peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } }
I20260925 10:39:16.628821 26218 leader_election.cc:288] T a997129f30954aaea55d2f8ce094a614 P b87295069e6d49a49c4e68fb5af6fce0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ae87327c48c44938d940dd9995a4d7e (127.23.100.1:42361), f216ed6b172f40999704a36c3b47bfdd (127.23.100.3:41541)
I20260925 10:39:16.629382 26147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a997129f30954aaea55d2f8ce094a614" candidate_uuid: "b87295069e6d49a49c4e68fb5af6fce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f216ed6b172f40999704a36c3b47bfdd" is_pre_election: true
I20260925 10:39:16.629402 25869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a997129f30954aaea55d2f8ce094a614" candidate_uuid: "b87295069e6d49a49c4e68fb5af6fce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ae87327c48c44938d940dd9995a4d7e" is_pre_election: true
I20260925 10:39:16.629678 26147 raft_consensus.cc:2454] T a997129f30954aaea55d2f8ce094a614 P f216ed6b172f40999704a36c3b47bfdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87295069e6d49a49c4e68fb5af6fce0 in term 0.
I20260925 10:39:16.629655 25869 raft_consensus.cc:2454] T a997129f30954aaea55d2f8ce094a614 P 5ae87327c48c44938d940dd9995a4d7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87295069e6d49a49c4e68fb5af6fce0 in term 0.
I20260925 10:39:16.630111 25938 leader_election.cc:302] T a997129f30954aaea55d2f8ce094a614 P b87295069e6d49a49c4e68fb5af6fce0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ae87327c48c44938d940dd9995a4d7e, b87295069e6d49a49c4e68fb5af6fce0; no voters:
I20260925 10:39:16.630468 26218 raft_consensus.cc:2781] T a997129f30954aaea55d2f8ce094a614 P b87295069e6d49a49c4e68fb5af6fce0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:39:16.630584 26218 raft_consensus.cc:482] T a997129f30954aaea55d2f8ce094a614 P b87295069e6d49a49c4e68fb5af6fce0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:39:16.630704 26218 raft_consensus.cc:3037] T a997129f30954aaea55d2f8ce094a614 P b87295069e6d49a49c4e68fb5af6fce0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:39:16.632599 26218 raft_consensus.cc:504] T a997129f30954aaea55d2f8ce094a614 P b87295069e6d49a49c4e68fb5af6fce0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } } peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } }
I20260925 10:39:16.633165 26218 leader_election.cc:288] T a997129f30954aaea55d2f8ce094a614 P b87295069e6d49a49c4e68fb5af6fce0 [CANDIDATE]: Term 1 election: Requested vote from peers 5ae87327c48c44938d940dd9995a4d7e (127.23.100.1:42361), f216ed6b172f40999704a36c3b47bfdd (127.23.100.3:41541)
I20260925 10:39:16.633522 25869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a997129f30954aaea55d2f8ce094a614" candidate_uuid: "b87295069e6d49a49c4e68fb5af6fce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ae87327c48c44938d940dd9995a4d7e"
I20260925 10:39:16.633594 26147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a997129f30954aaea55d2f8ce094a614" candidate_uuid: "b87295069e6d49a49c4e68fb5af6fce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f216ed6b172f40999704a36c3b47bfdd"
I20260925 10:39:16.633739 25869 raft_consensus.cc:3037] T a997129f30954aaea55d2f8ce094a614 P 5ae87327c48c44938d940dd9995a4d7e [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:39:16.633798 26147 raft_consensus.cc:3037] T a997129f30954aaea55d2f8ce094a614 P f216ed6b172f40999704a36c3b47bfdd [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:39:16.635747 26147 raft_consensus.cc:2454] T a997129f30954aaea55d2f8ce094a614 P f216ed6b172f40999704a36c3b47bfdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87295069e6d49a49c4e68fb5af6fce0 in term 1.
I20260925 10:39:16.635768 25869 raft_consensus.cc:2454] T a997129f30954aaea55d2f8ce094a614 P 5ae87327c48c44938d940dd9995a4d7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87295069e6d49a49c4e68fb5af6fce0 in term 1.
I20260925 10:39:16.636134 25938 leader_election.cc:302] T a997129f30954aaea55d2f8ce094a614 P b87295069e6d49a49c4e68fb5af6fce0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ae87327c48c44938d940dd9995a4d7e, b87295069e6d49a49c4e68fb5af6fce0; no voters:
I20260925 10:39:16.636449 26218 raft_consensus.cc:2781] T a997129f30954aaea55d2f8ce094a614 P b87295069e6d49a49c4e68fb5af6fce0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:39:16.636624 26218 raft_consensus.cc:686] T a997129f30954aaea55d2f8ce094a614 P b87295069e6d49a49c4e68fb5af6fce0 [term 1 LEADER]: Becoming Leader. State: Replica: b87295069e6d49a49c4e68fb5af6fce0, State: Running, Role: LEADER
I20260925 10:39:16.636855 26218 consensus_queue.cc:237] T a997129f30954aaea55d2f8ce094a614 P b87295069e6d49a49c4e68fb5af6fce0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } } peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } }
I20260925 10:39:16.638623 26224 tablet_replica.cc:439] T a997129f30954aaea55d2f8ce094a614 P b87295069e6d49a49c4e68fb5af6fce0: TxnStatusTablet state changed. Reason: New leader b87295069e6d49a49c4e68fb5af6fce0. Latest consensus state: current_term: 1 leader_uuid: "b87295069e6d49a49c4e68fb5af6fce0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } } peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } } }
I20260925 10:39:16.639016 26224 tablet_replica.cc:442] T a997129f30954aaea55d2f8ce094a614 P b87295069e6d49a49c4e68fb5af6fce0: This TxnStatusTablet replica's current role is: LEADER
I20260925 10:39:16.639700 26236 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 10:39:16.639962 26236 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 10:39:16.640064 25707 catalog_manager.cc:5495] T a997129f30954aaea55d2f8ce094a614 P b87295069e6d49a49c4e68fb5af6fce0 reported cstate change: term changed from 0 to 1, leader changed from <none> to b87295069e6d49a49c4e68fb5af6fce0 (127.23.100.2). New cstate: current_term: 1 leader_uuid: "b87295069e6d49a49c4e68fb5af6fce0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } health_report { overall_health: UNKNOWN } } }
I20260925 10:39:16.667299 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:16.661279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53276 (local address 127.23.100.2:33655)
0925 10:39:16.661469 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:16.661475 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:16.661601 (+ 126us) server_negotiation.cc:407] Connection header received
0925 10:39:16.661861 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:16.661864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:16.661939 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:16.662064 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:39:16.662492 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:16.663209 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:16.664922 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:16.665450 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:16.665546 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:16.666076 (+ 530us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:39:16.666615 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:39:16.666733 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:16.666908 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 10:39:16.666972 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 10:39:16.667351 26231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:16.661262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.2:33655 (local address 127.0.0.1:53276)
0925 10:39:16.661507 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:16.661527 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:16.661673 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:16.662117 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:16.662122 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:16.662135 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:39:16.662365 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:16.662380 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:16.663329 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:16.663334 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:16.664799 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:16.664814 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:16.665630 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:16.665636 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:16.665849 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:16.665920 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:39:16.666729 (+ 809us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:39:16.666749 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 10:39:16.666859 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 10:39:16.667010 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 10:39:16.674265 25869 raft_consensus.cc:1261] T a997129f30954aaea55d2f8ce094a614 P 5ae87327c48c44938d940dd9995a4d7e [term 1 FOLLOWER]: Refusing update from remote peer b87295069e6d49a49c4e68fb5af6fce0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:39:16.674634 26147 raft_consensus.cc:1261] T a997129f30954aaea55d2f8ce094a614 P f216ed6b172f40999704a36c3b47bfdd [term 1 FOLLOWER]: Refusing update from remote peer b87295069e6d49a49c4e68fb5af6fce0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:39:16.675149 26224 consensus_queue.cc:1035] T a997129f30954aaea55d2f8ce094a614 P b87295069e6d49a49c4e68fb5af6fce0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:39:16.675459 26218 consensus_queue.cc:1035] T a997129f30954aaea55d2f8ce094a614 P b87295069e6d49a49c4e68fb5af6fce0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:39:16.682575 26220 tablet_replica.cc:439] T a997129f30954aaea55d2f8ce094a614 P f216ed6b172f40999704a36c3b47bfdd: TxnStatusTablet state changed. Reason: New leader b87295069e6d49a49c4e68fb5af6fce0. Latest consensus state: current_term: 1 leader_uuid: "b87295069e6d49a49c4e68fb5af6fce0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } } peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } } }
I20260925 10:39:16.682879 26220 tablet_replica.cc:442] T a997129f30954aaea55d2f8ce094a614 P f216ed6b172f40999704a36c3b47bfdd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:39:16.685747 26239 mvcc.cc:205] Tried to move back new op lower bound from 7333202971329224704 to 7333202971186929664. Current Snapshot: MvccSnapshot[applied={T|T < 7333202971329224704}]
I20260925 10:39:16.686800 26218 tablet_replica.cc:439] T a997129f30954aaea55d2f8ce094a614 P b87295069e6d49a49c4e68fb5af6fce0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b87295069e6d49a49c4e68fb5af6fce0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } } peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } } }
I20260925 10:39:16.687140 26218 tablet_replica.cc:442] T a997129f30954aaea55d2f8ce094a614 P b87295069e6d49a49c4e68fb5af6fce0: This TxnStatusTablet replica's current role is: LEADER
I20260925 10:39:16.687954 26240 mvcc.cc:205] Tried to move back new op lower bound from 7333202971329224704 to 7333202971186929664. Current Snapshot: MvccSnapshot[applied={T|T < 7333202971329224704}]
I20260925 10:39:16.691382 26220 tablet_replica.cc:439] T a997129f30954aaea55d2f8ce094a614 P f216ed6b172f40999704a36c3b47bfdd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b87295069e6d49a49c4e68fb5af6fce0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } } peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } } }
I20260925 10:39:16.691636 26220 tablet_replica.cc:442] T a997129f30954aaea55d2f8ce094a614 P f216ed6b172f40999704a36c3b47bfdd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:39:16.695480 26219 tablet_replica.cc:439] T a997129f30954aaea55d2f8ce094a614 P 5ae87327c48c44938d940dd9995a4d7e: TxnStatusTablet state changed. Reason: New leader b87295069e6d49a49c4e68fb5af6fce0. Latest consensus state: current_term: 1 leader_uuid: "b87295069e6d49a49c4e68fb5af6fce0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } } peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } } }
I20260925 10:39:16.695791 26219 tablet_replica.cc:442] T a997129f30954aaea55d2f8ce094a614 P 5ae87327c48c44938d940dd9995a4d7e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:39:16.697695 26218 tablet_replica.cc:439] T a997129f30954aaea55d2f8ce094a614 P b87295069e6d49a49c4e68fb5af6fce0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b87295069e6d49a49c4e68fb5af6fce0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } } peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } } }
I20260925 10:39:16.697955 26218 tablet_replica.cc:442] T a997129f30954aaea55d2f8ce094a614 P b87295069e6d49a49c4e68fb5af6fce0: This TxnStatusTablet replica's current role is: LEADER
I20260925 10:39:16.704660 26219 tablet_replica.cc:439] T a997129f30954aaea55d2f8ce094a614 P 5ae87327c48c44938d940dd9995a4d7e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b87295069e6d49a49c4e68fb5af6fce0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } } peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } } }
I20260925 10:39:16.704896 26219 tablet_replica.cc:442] T a997129f30954aaea55d2f8ce094a614 P 5ae87327c48c44938d940dd9995a4d7e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:39:16.712553 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:16.706697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53290 (local address 127.23.100.2:33655)
0925 10:39:16.706887 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:16.706895 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:16.707404 (+ 509us) server_negotiation.cc:407] Connection header received
0925 10:39:16.707725 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:16.707730 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:16.707826 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:16.707939 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:39:16.708380 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:16.709259 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:16.710862 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:16.711266 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:16.711432 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:16.711915 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:39:16.712016 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:39:16.712084 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:16.712235 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 10:39:16.712294 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 10:39:16.729195 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:16.722866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:52327 (local address 127.23.100.2:33655)
0925 10:39:16.723044 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:16.723051 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:16.723664 (+ 613us) server_negotiation.cc:407] Connection header received
0925 10:39:16.723834 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:16.723837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:16.723890 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:16.723957 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:39:16.724471 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:16.725206 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:16.727350 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:16.727934 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:16.728026 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:16.728565 (+ 539us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:39:16.728666 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:39:16.728732 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:16.728881 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 10:39:16.728934 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 10:39:16.729573 26251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:16.722912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.2:33655 (local address 127.23.100.2:52327)
0925 10:39:16.723568 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:16.723586 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:16.723713 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:16.723995 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:16.724000 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:16.724012 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:39:16.724322 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:16.724333 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:16.725351 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:16.725357 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:16.727199 (+ 1842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:16.727213 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:16.728118 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:16.728125 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:16.728305 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:16.728370 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:39:16.728759 (+ 389us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:39:16.728777 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 10:39:16.728858 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 10:39:16.728994 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":346,"threads_started":1}
I20260925 10:39:16.739705 25869 raft_consensus.cc:1261] T a6cfbc1f6f664e65bde7a073df41da39 P 5ae87327c48c44938d940dd9995a4d7e [term 1 FOLLOWER]: Refusing update from remote peer b87295069e6d49a49c4e68fb5af6fce0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:39:16.739919 26147 raft_consensus.cc:1261] T a6cfbc1f6f664e65bde7a073df41da39 P f216ed6b172f40999704a36c3b47bfdd [term 1 FOLLOWER]: Refusing update from remote peer b87295069e6d49a49c4e68fb5af6fce0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:39:16.740290 26252 consensus_queue.cc:1035] T a6cfbc1f6f664e65bde7a073df41da39 P b87295069e6d49a49c4e68fb5af6fce0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:39:16.740782 26218 consensus_queue.cc:1035] T a6cfbc1f6f664e65bde7a073df41da39 P b87295069e6d49a49c4e68fb5af6fce0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:39:16.746230 26241 mvcc.cc:205] Tried to move back new op lower bound from 7333202971600482304 to 7333202970510876672. Current Snapshot: MvccSnapshot[applied={T|T < 7333202971600482304}]
I20260925 10:39:16.807251 25707 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:39:16.807530 25707 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:39:16.811154 25707 catalog_manager.cc:5782] T 00000000000000000000000000000000 P beeffe7071f041e297918aa03aa556c9: Sending DeleteTablet for 3 replicas of tablet a6cfbc1f6f664e65bde7a073df41da39
I20260925 10:39:16.812376 25845 tablet_service.cc:1514] Processing DeleteTablet for tablet a6cfbc1f6f664e65bde7a073df41da39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:39:16 UTC) from {username='kudu', principal='kudu/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:33618
I20260925 10:39:16.812695 25984 tablet_service.cc:1514] Processing DeleteTablet for tablet a6cfbc1f6f664e65bde7a073df41da39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:39:16 UTC) from {username='kudu', principal='kudu/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:53270
I20260925 10:39:16.812721 26123 tablet_service.cc:1514] Processing DeleteTablet for tablet a6cfbc1f6f664e65bde7a073df41da39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:39:16 UTC) from {username='kudu', principal='kudu/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:35528
I20260925 10:39:16.814531 25707 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43768:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:39:16.815385 25707 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:39:16.823364 26257 tablet_replica.cc:330] T a6cfbc1f6f664e65bde7a073df41da39 P b87295069e6d49a49c4e68fb5af6fce0: stopping tablet replica
I20260925 10:39:16.824168 26257 raft_consensus.cc:2229] T a6cfbc1f6f664e65bde7a073df41da39 P b87295069e6d49a49c4e68fb5af6fce0 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:39:16.825142 26257 raft_consensus.cc:2258] T a6cfbc1f6f664e65bde7a073df41da39 P b87295069e6d49a49c4e68fb5af6fce0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:39:16.826061 26122 tablet_service.cc:1467] Processing CreateTablet for tablet 8e4b2985997842c883765cc230934e6f (DEFAULT_TABLE table=test-table [id=59ad3c4beb134c428a608c37313f2b2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:39:16.826418 25844 tablet_service.cc:1467] Processing CreateTablet for tablet 8e4b2985997842c883765cc230934e6f (DEFAULT_TABLE table=test-table [id=59ad3c4beb134c428a608c37313f2b2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:39:16.826670 26122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e4b2985997842c883765cc230934e6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:39:16.826994 25844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e4b2985997842c883765cc230934e6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:39:16.834760 26215 tablet_bootstrap.cc:492] T 8e4b2985997842c883765cc230934e6f P f216ed6b172f40999704a36c3b47bfdd: Bootstrap starting.
I20260925 10:39:16.836619 26215 tablet_bootstrap.cc:654] T 8e4b2985997842c883765cc230934e6f P f216ed6b172f40999704a36c3b47bfdd: Neither blocks nor log segments found. Creating new log.
I20260925 10:39:16.839139 26214 tablet_bootstrap.cc:492] T 8e4b2985997842c883765cc230934e6f P 5ae87327c48c44938d940dd9995a4d7e: Bootstrap starting.
I20260925 10:39:16.830715 25984 tablet_service.cc:1467] Processing CreateTablet for tablet 8e4b2985997842c883765cc230934e6f (DEFAULT_TABLE table=test-table [id=59ad3c4beb134c428a608c37313f2b2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:39:16.841569 26214 tablet_bootstrap.cc:654] T 8e4b2985997842c883765cc230934e6f P 5ae87327c48c44938d940dd9995a4d7e: Neither blocks nor log segments found. Creating new log.
I20260925 10:39:16.841954 25984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e4b2985997842c883765cc230934e6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:39:16.845770 26256 tablet_replica.cc:330] T a6cfbc1f6f664e65bde7a073df41da39 P 5ae87327c48c44938d940dd9995a4d7e: stopping tablet replica
I20260925 10:39:16.846283 26256 raft_consensus.cc:2229] T a6cfbc1f6f664e65bde7a073df41da39 P 5ae87327c48c44938d940dd9995a4d7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:39:16.846751 26256 raft_consensus.cc:2258] T a6cfbc1f6f664e65bde7a073df41da39 P 5ae87327c48c44938d940dd9995a4d7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:39:16.847551 26258 tablet_replica.cc:330] T a6cfbc1f6f664e65bde7a073df41da39 P f216ed6b172f40999704a36c3b47bfdd: stopping tablet replica
I20260925 10:39:16.848024 26258 raft_consensus.cc:2229] T a6cfbc1f6f664e65bde7a073df41da39 P f216ed6b172f40999704a36c3b47bfdd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:39:16.850113 26256 ts_tablet_manager.cc:1905] T a6cfbc1f6f664e65bde7a073df41da39 P 5ae87327c48c44938d940dd9995a4d7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:39:16.851735 26214 tablet_bootstrap.cc:492] T 8e4b2985997842c883765cc230934e6f P 5ae87327c48c44938d940dd9995a4d7e: No bootstrap required, opened a new log
I20260925 10:39:16.851972 26214 ts_tablet_manager.cc:1397] T 8e4b2985997842c883765cc230934e6f P 5ae87327c48c44938d940dd9995a4d7e: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.004s
I20260925 10:39:16.852034 26258 raft_consensus.cc:2258] T a6cfbc1f6f664e65bde7a073df41da39 P f216ed6b172f40999704a36c3b47bfdd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:39:16.852293 26257 ts_tablet_manager.cc:1905] T a6cfbc1f6f664e65bde7a073df41da39 P b87295069e6d49a49c4e68fb5af6fce0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:39:16.852772 26214 raft_consensus.cc:348] T 8e4b2985997842c883765cc230934e6f P 5ae87327c48c44938d940dd9995a4d7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } } peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } }
I20260925 10:39:16.853014 26214 raft_consensus.cc:374] T 8e4b2985997842c883765cc230934e6f P 5ae87327c48c44938d940dd9995a4d7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:39:16.853085 26214 raft_consensus.cc:729] T 8e4b2985997842c883765cc230934e6f P 5ae87327c48c44938d940dd9995a4d7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ae87327c48c44938d940dd9995a4d7e, State: Initialized, Role: FOLLOWER
I20260925 10:39:16.854749 26214 consensus_queue.cc:260] T 8e4b2985997842c883765cc230934e6f P 5ae87327c48c44938d940dd9995a4d7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } } peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } }
I20260925 10:39:16.855487 26214 ts_tablet_manager.cc:1428] T 8e4b2985997842c883765cc230934e6f P 5ae87327c48c44938d940dd9995a4d7e: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20260925 10:39:16.855922 26258 ts_tablet_manager.cc:1905] T a6cfbc1f6f664e65bde7a073df41da39 P f216ed6b172f40999704a36c3b47bfdd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:39:16.859725 26212 tablet_bootstrap.cc:492] T 8e4b2985997842c883765cc230934e6f P b87295069e6d49a49c4e68fb5af6fce0: Bootstrap starting.
I20260925 10:39:16.861562 26212 tablet_bootstrap.cc:654] T 8e4b2985997842c883765cc230934e6f P b87295069e6d49a49c4e68fb5af6fce0: Neither blocks nor log segments found. Creating new log.
I20260925 10:39:16.865957 26256 ts_tablet_manager.cc:1918] T a6cfbc1f6f664e65bde7a073df41da39 P 5ae87327c48c44938d940dd9995a4d7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:39:16.866189 26256 log.cc:1196] T a6cfbc1f6f664e65bde7a073df41da39 P 5ae87327c48c44938d940dd9995a4d7e: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-0/wal/wals/a6cfbc1f6f664e65bde7a073df41da39
I20260925 10:39:16.867092 26258 ts_tablet_manager.cc:1918] T a6cfbc1f6f664e65bde7a073df41da39 P f216ed6b172f40999704a36c3b47bfdd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:39:16.867280 26258 log.cc:1196] T a6cfbc1f6f664e65bde7a073df41da39 P f216ed6b172f40999704a36c3b47bfdd: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-2/wal/wals/a6cfbc1f6f664e65bde7a073df41da39
I20260925 10:39:16.867862 26258 ts_tablet_manager.cc:1939] T a6cfbc1f6f664e65bde7a073df41da39 P f216ed6b172f40999704a36c3b47bfdd: Deleting consensus metadata
I20260925 10:39:16.868237 26256 ts_tablet_manager.cc:1939] T a6cfbc1f6f664e65bde7a073df41da39 P 5ae87327c48c44938d940dd9995a4d7e: Deleting consensus metadata
I20260925 10:39:16.870047 25696 catalog_manager.cc:4841] TS 5ae87327c48c44938d940dd9995a4d7e (127.23.100.1:42361): tablet a6cfbc1f6f664e65bde7a073df41da39 (table test-table [id=cdc555a62b304e51b28550247bc669ef]) successfully deleted
I20260925 10:39:16.870460 25695 catalog_manager.cc:4841] TS f216ed6b172f40999704a36c3b47bfdd (127.23.100.3:41541): tablet a6cfbc1f6f664e65bde7a073df41da39 (table test-table [id=cdc555a62b304e51b28550247bc669ef]) successfully deleted
I20260925 10:39:16.873104 26212 tablet_bootstrap.cc:492] T 8e4b2985997842c883765cc230934e6f P b87295069e6d49a49c4e68fb5af6fce0: No bootstrap required, opened a new log
I20260925 10:39:16.873301 26212 ts_tablet_manager.cc:1397] T 8e4b2985997842c883765cc230934e6f P b87295069e6d49a49c4e68fb5af6fce0: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.001s
I20260925 10:39:16.874115 26212 raft_consensus.cc:348] T 8e4b2985997842c883765cc230934e6f P b87295069e6d49a49c4e68fb5af6fce0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } } peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } }
I20260925 10:39:16.874362 26212 raft_consensus.cc:374] T 8e4b2985997842c883765cc230934e6f P b87295069e6d49a49c4e68fb5af6fce0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:39:16.874467 26212 raft_consensus.cc:729] T 8e4b2985997842c883765cc230934e6f P b87295069e6d49a49c4e68fb5af6fce0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87295069e6d49a49c4e68fb5af6fce0, State: Initialized, Role: FOLLOWER
I20260925 10:39:16.874747 26212 consensus_queue.cc:260] T 8e4b2985997842c883765cc230934e6f P b87295069e6d49a49c4e68fb5af6fce0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } } peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } }
I20260925 10:39:16.875347 26212 ts_tablet_manager.cc:1428] T 8e4b2985997842c883765cc230934e6f P b87295069e6d49a49c4e68fb5af6fce0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:39:16.880282 26257 ts_tablet_manager.cc:1918] T a6cfbc1f6f664e65bde7a073df41da39 P b87295069e6d49a49c4e68fb5af6fce0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:39:16.880475 26257 log.cc:1196] T a6cfbc1f6f664e65bde7a073df41da39 P b87295069e6d49a49c4e68fb5af6fce0: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-1/wal/wals/a6cfbc1f6f664e65bde7a073df41da39
I20260925 10:39:16.880484 26215 tablet_bootstrap.cc:492] T 8e4b2985997842c883765cc230934e6f P f216ed6b172f40999704a36c3b47bfdd: No bootstrap required, opened a new log
I20260925 10:39:16.880662 26215 ts_tablet_manager.cc:1397] T 8e4b2985997842c883765cc230934e6f P f216ed6b172f40999704a36c3b47bfdd: Time spent bootstrapping tablet: real 0.046s user 0.009s sys 0.003s
I20260925 10:39:16.881079 26257 ts_tablet_manager.cc:1939] T a6cfbc1f6f664e65bde7a073df41da39 P b87295069e6d49a49c4e68fb5af6fce0: Deleting consensus metadata
I20260925 10:39:16.881516 26215 raft_consensus.cc:348] T 8e4b2985997842c883765cc230934e6f P f216ed6b172f40999704a36c3b47bfdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } } peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } }
I20260925 10:39:16.881764 26215 raft_consensus.cc:374] T 8e4b2985997842c883765cc230934e6f P f216ed6b172f40999704a36c3b47bfdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:39:16.881850 26215 raft_consensus.cc:729] T 8e4b2985997842c883765cc230934e6f P f216ed6b172f40999704a36c3b47bfdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f216ed6b172f40999704a36c3b47bfdd, State: Initialized, Role: FOLLOWER
I20260925 10:39:16.882112 26215 consensus_queue.cc:260] T 8e4b2985997842c883765cc230934e6f P f216ed6b172f40999704a36c3b47bfdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } } peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } }
I20260925 10:39:16.882867 26215 ts_tablet_manager.cc:1428] T 8e4b2985997842c883765cc230934e6f P f216ed6b172f40999704a36c3b47bfdd: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 10:39:16.882956 25695 catalog_manager.cc:4841] TS b87295069e6d49a49c4e68fb5af6fce0 (127.23.100.2:33655): tablet a6cfbc1f6f664e65bde7a073df41da39 (table test-table [id=cdc555a62b304e51b28550247bc669ef]) successfully deleted
I20260925 10:39:17.088256 26220 raft_consensus.cc:482] T 8e4b2985997842c883765cc230934e6f P f216ed6b172f40999704a36c3b47bfdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:39:17.088450 26220 raft_consensus.cc:504] T 8e4b2985997842c883765cc230934e6f P f216ed6b172f40999704a36c3b47bfdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } } peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } }
I20260925 10:39:17.089152 26220 leader_election.cc:288] T 8e4b2985997842c883765cc230934e6f P f216ed6b172f40999704a36c3b47bfdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ae87327c48c44938d940dd9995a4d7e (127.23.100.1:42361), b87295069e6d49a49c4e68fb5af6fce0 (127.23.100.2:33655)
I20260925 10:39:17.089735 25869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4b2985997842c883765cc230934e6f" candidate_uuid: "f216ed6b172f40999704a36c3b47bfdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ae87327c48c44938d940dd9995a4d7e" is_pre_election: true
I20260925 10:39:17.089793 26006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4b2985997842c883765cc230934e6f" candidate_uuid: "f216ed6b172f40999704a36c3b47bfdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87295069e6d49a49c4e68fb5af6fce0" is_pre_election: true
I20260925 10:39:17.089986 25869 raft_consensus.cc:2454] T 8e4b2985997842c883765cc230934e6f P 5ae87327c48c44938d940dd9995a4d7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f216ed6b172f40999704a36c3b47bfdd in term 0.
I20260925 10:39:17.090126 26006 raft_consensus.cc:2454] T 8e4b2985997842c883765cc230934e6f P b87295069e6d49a49c4e68fb5af6fce0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f216ed6b172f40999704a36c3b47bfdd in term 0.
I20260925 10:39:17.090729 26076 leader_election.cc:302] T 8e4b2985997842c883765cc230934e6f P f216ed6b172f40999704a36c3b47bfdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b87295069e6d49a49c4e68fb5af6fce0, f216ed6b172f40999704a36c3b47bfdd; no voters:
I20260925 10:39:17.091066 26220 raft_consensus.cc:2781] T 8e4b2985997842c883765cc230934e6f P f216ed6b172f40999704a36c3b47bfdd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:39:17.091193 26220 raft_consensus.cc:482] T 8e4b2985997842c883765cc230934e6f P f216ed6b172f40999704a36c3b47bfdd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:39:17.091308 26220 raft_consensus.cc:3037] T 8e4b2985997842c883765cc230934e6f P f216ed6b172f40999704a36c3b47bfdd [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:39:17.094256 26220 raft_consensus.cc:504] T 8e4b2985997842c883765cc230934e6f P f216ed6b172f40999704a36c3b47bfdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } } peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } }
I20260925 10:39:17.095440 25869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4b2985997842c883765cc230934e6f" candidate_uuid: "f216ed6b172f40999704a36c3b47bfdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ae87327c48c44938d940dd9995a4d7e"
I20260925 10:39:17.095446 26006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4b2985997842c883765cc230934e6f" candidate_uuid: "f216ed6b172f40999704a36c3b47bfdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87295069e6d49a49c4e68fb5af6fce0"
I20260925 10:39:17.095677 26006 raft_consensus.cc:3037] T 8e4b2985997842c883765cc230934e6f P b87295069e6d49a49c4e68fb5af6fce0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:39:17.095675 25869 raft_consensus.cc:3037] T 8e4b2985997842c883765cc230934e6f P 5ae87327c48c44938d940dd9995a4d7e [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:39:17.098559 26006 raft_consensus.cc:2454] T 8e4b2985997842c883765cc230934e6f P b87295069e6d49a49c4e68fb5af6fce0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f216ed6b172f40999704a36c3b47bfdd in term 1.
I20260925 10:39:17.098664 25869 raft_consensus.cc:2454] T 8e4b2985997842c883765cc230934e6f P 5ae87327c48c44938d940dd9995a4d7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f216ed6b172f40999704a36c3b47bfdd in term 1.
I20260925 10:39:17.099073 26076 leader_election.cc:302] T 8e4b2985997842c883765cc230934e6f P f216ed6b172f40999704a36c3b47bfdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b87295069e6d49a49c4e68fb5af6fce0, f216ed6b172f40999704a36c3b47bfdd; no voters:
I20260925 10:39:17.099618 26220 leader_election.cc:288] T 8e4b2985997842c883765cc230934e6f P f216ed6b172f40999704a36c3b47bfdd [CANDIDATE]: Term 1 election: Requested vote from peers 5ae87327c48c44938d940dd9995a4d7e (127.23.100.1:42361), b87295069e6d49a49c4e68fb5af6fce0 (127.23.100.2:33655)
I20260925 10:39:17.099700 26262 raft_consensus.cc:2781] T 8e4b2985997842c883765cc230934e6f P f216ed6b172f40999704a36c3b47bfdd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:39:17.100397 26262 raft_consensus.cc:686] T 8e4b2985997842c883765cc230934e6f P f216ed6b172f40999704a36c3b47bfdd [term 1 LEADER]: Becoming Leader. State: Replica: f216ed6b172f40999704a36c3b47bfdd, State: Running, Role: LEADER
I20260925 10:39:17.100818 26262 consensus_queue.cc:237] T 8e4b2985997842c883765cc230934e6f P f216ed6b172f40999704a36c3b47bfdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } } peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } }
I20260925 10:39:17.104615 25706 catalog_manager.cc:5495] T 8e4b2985997842c883765cc230934e6f P f216ed6b172f40999704a36c3b47bfdd reported cstate change: term changed from 0 to 1, leader changed from <none> to f216ed6b172f40999704a36c3b47bfdd (127.23.100.3). New cstate: current_term: 1 leader_uuid: "f216ed6b172f40999704a36c3b47bfdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } health_report { overall_health: HEALTHY } } }
I20260925 10:39:17.151847 26265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:17.144931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35542 (local address 127.23.100.3:41541)
0925 10:39:17.145735 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:17.145739 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:17.145754 (+ 15us) server_negotiation.cc:407] Connection header received
0925 10:39:17.145833 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:17.145837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:17.145907 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:17.146055 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:39:17.146526 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:17.147277 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:17.149238 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:17.149715 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:17.149837 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:17.150367 (+ 530us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:39:17.151058 (+ 691us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:39:17.151181 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:17.151299 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 10:39:17.151365 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":694,"thread_start_us":560,"threads_started":1}
I20260925 10:39:17.168193 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:17.162692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:46369 (local address 127.23.100.2:33655)
0925 10:39:17.162872 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:17.162880 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:17.163370 (+ 490us) server_negotiation.cc:407] Connection header received
0925 10:39:17.163587 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:17.163595 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:17.163657 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:17.163751 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:39:17.164218 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:17.164981 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:17.166440 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:17.166884 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:17.167019 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:17.167454 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:39:17.167572 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:39:17.167674 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:17.167806 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 10:39:17.167883 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 10:39:17.168306 26268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:17.162666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.2:33655 (local address 127.23.100.3:46369)
0925 10:39:17.163269 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:17.163284 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:17.163440 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:17.163763 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:17.163768 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:17.163778 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:39:17.164080 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:17.164089 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:17.165106 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:17.165114 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:17.166285 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:17.166296 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:17.167054 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:17.167061 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:17.167203 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:17.167256 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:39:17.167673 (+ 417us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:39:17.167686 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 10:39:17.167782 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 10:39:17.167913 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":317,"threads_started":1}
I20260925 10:39:17.178490 25869 raft_consensus.cc:1261] T 8e4b2985997842c883765cc230934e6f P 5ae87327c48c44938d940dd9995a4d7e [term 1 FOLLOWER]: Refusing update from remote peer f216ed6b172f40999704a36c3b47bfdd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:39:17.178627 26006 raft_consensus.cc:1261] T 8e4b2985997842c883765cc230934e6f P b87295069e6d49a49c4e68fb5af6fce0 [term 1 FOLLOWER]: Refusing update from remote peer f216ed6b172f40999704a36c3b47bfdd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:39:17.179080 26263 consensus_queue.cc:1035] T 8e4b2985997842c883765cc230934e6f P f216ed6b172f40999704a36c3b47bfdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:39:17.179533 26262 consensus_queue.cc:1035] T 8e4b2985997842c883765cc230934e6f P f216ed6b172f40999704a36c3b47bfdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:39:17.211678 26265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:17.205584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:33293 (local address 127.23.100.3:41541)
0925 10:39:17.205731 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:17.205736 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:17.205752 (+ 16us) server_negotiation.cc:407] Connection header received
0925 10:39:17.205916 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:17.205921 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:17.206003 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:17.206093 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:39:17.206682 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:17.207661 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:17.209646 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:17.210333 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:17.210469 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:17.210929 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:39:17.211073 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:39:17.211168 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:17.211326 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 10:39:17.211400 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 10:39:17.211724 26251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:17.205462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.3:41541 (local address 127.23.100.2:33293)
0925 10:39:17.205630 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:17.205650 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:17.205758 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:17.206264 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:17.206270 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:17.206282 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:39:17.206517 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:17.206530 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:17.207852 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:17.207859 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:17.209490 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:17.209504 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:17.210488 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:17.210493 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:17.210670 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:17.210757 (+ 87us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:39:17.211174 (+ 417us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:39:17.211184 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 10:39:17.211301 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 10:39:17.211435 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 10:39:17.212792 26240 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 10:39:18.802500 25707 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:39:18.802722 25707 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:39:18.806995 25707 catalog_manager.cc:5782] T 00000000000000000000000000000000 P beeffe7071f041e297918aa03aa556c9: Sending DeleteTablet for 3 replicas of tablet 8e4b2985997842c883765cc230934e6f
I20260925 10:39:18.807734 25845 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e4b2985997842c883765cc230934e6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:39:18 UTC) from {username='kudu', principal='kudu/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:33618
I20260925 10:39:18.807861 25983 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e4b2985997842c883765cc230934e6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:39:18 UTC) from {username='kudu', principal='kudu/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:53270
I20260925 10:39:18.807855 26123 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e4b2985997842c883765cc230934e6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:39:18 UTC) from {username='kudu', principal='kudu/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:35528
I20260925 10:39:18.820253 26294 tablet_replica.cc:330] T 8e4b2985997842c883765cc230934e6f P 5ae87327c48c44938d940dd9995a4d7e: stopping tablet replica
I20260925 10:39:18.820492 26295 tablet_replica.cc:330] T 8e4b2985997842c883765cc230934e6f P b87295069e6d49a49c4e68fb5af6fce0: stopping tablet replica
I20260925 10:39:18.820598 26296 tablet_replica.cc:330] T 8e4b2985997842c883765cc230934e6f P f216ed6b172f40999704a36c3b47bfdd: stopping tablet replica
I20260925 10:39:18.820639 26294 raft_consensus.cc:2229] T 8e4b2985997842c883765cc230934e6f P 5ae87327c48c44938d940dd9995a4d7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:39:18.820791 26295 raft_consensus.cc:2229] T 8e4b2985997842c883765cc230934e6f P b87295069e6d49a49c4e68fb5af6fce0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:39:18.820914 26294 raft_consensus.cc:2258] T 8e4b2985997842c883765cc230934e6f P 5ae87327c48c44938d940dd9995a4d7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:39:18.820873 26296 raft_consensus.cc:2229] T 8e4b2985997842c883765cc230934e6f P f216ed6b172f40999704a36c3b47bfdd [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:39:18.821097 26295 raft_consensus.cc:2258] T 8e4b2985997842c883765cc230934e6f P b87295069e6d49a49c4e68fb5af6fce0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:39:18.821455 26296 raft_consensus.cc:2258] T 8e4b2985997842c883765cc230934e6f P f216ed6b172f40999704a36c3b47bfdd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:39:18.822032 26294 ts_tablet_manager.cc:1905] T 8e4b2985997842c883765cc230934e6f P 5ae87327c48c44938d940dd9995a4d7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:39:18.822237 26295 ts_tablet_manager.cc:1905] T 8e4b2985997842c883765cc230934e6f P b87295069e6d49a49c4e68fb5af6fce0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:39:18.822372 26296 ts_tablet_manager.cc:1905] T 8e4b2985997842c883765cc230934e6f P f216ed6b172f40999704a36c3b47bfdd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:39:18.828631 26294 ts_tablet_manager.cc:1918] T 8e4b2985997842c883765cc230934e6f P 5ae87327c48c44938d940dd9995a4d7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 10:39:18.828819 26294 log.cc:1196] T 8e4b2985997842c883765cc230934e6f P 5ae87327c48c44938d940dd9995a4d7e: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-0/wal/wals/8e4b2985997842c883765cc230934e6f
I20260925 10:39:18.829181 26296 ts_tablet_manager.cc:1918] T 8e4b2985997842c883765cc230934e6f P f216ed6b172f40999704a36c3b47bfdd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 10:39:18.829352 26294 ts_tablet_manager.cc:1939] T 8e4b2985997842c883765cc230934e6f P 5ae87327c48c44938d940dd9995a4d7e: Deleting consensus metadata
I20260925 10:39:18.829357 26296 log.cc:1196] T 8e4b2985997842c883765cc230934e6f P f216ed6b172f40999704a36c3b47bfdd: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-2/wal/wals/8e4b2985997842c883765cc230934e6f
I20260925 10:39:18.829838 26296 ts_tablet_manager.cc:1939] T 8e4b2985997842c883765cc230934e6f P f216ed6b172f40999704a36c3b47bfdd: Deleting consensus metadata
I20260925 10:39:18.829895 26295 ts_tablet_manager.cc:1918] T 8e4b2985997842c883765cc230934e6f P b87295069e6d49a49c4e68fb5af6fce0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 10:39:18.830112 26295 log.cc:1196] T 8e4b2985997842c883765cc230934e6f P b87295069e6d49a49c4e68fb5af6fce0: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-1/wal/wals/8e4b2985997842c883765cc230934e6f
I20260925 10:39:18.830651 26295 ts_tablet_manager.cc:1939] T 8e4b2985997842c883765cc230934e6f P b87295069e6d49a49c4e68fb5af6fce0: Deleting consensus metadata
I20260925 10:39:18.830672 25696 catalog_manager.cc:4841] TS 5ae87327c48c44938d940dd9995a4d7e (127.23.100.1:42361): tablet 8e4b2985997842c883765cc230934e6f (table test-table [id=59ad3c4beb134c428a608c37313f2b2e]) successfully deleted
I20260925 10:39:18.831415 25695 catalog_manager.cc:4841] TS f216ed6b172f40999704a36c3b47bfdd (127.23.100.3:41541): tablet 8e4b2985997842c883765cc230934e6f (table test-table [id=59ad3c4beb134c428a608c37313f2b2e]) successfully deleted
I20260925 10:39:18.831770 25695 catalog_manager.cc:4841] TS b87295069e6d49a49c4e68fb5af6fce0 (127.23.100.2:33655): tablet 8e4b2985997842c883765cc230934e6f (table test-table [id=59ad3c4beb134c428a608c37313f2b2e]) successfully deleted
Sep 25 10:39:18 dist-test-slave-dxdd krb5kdc[25664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332758, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 10:39:18 dist-test-slave-dxdd krb5kdc[25664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332758, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:39:18.865427 26304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:18.852215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43782 (local address 127.23.100.62:35469)
0925 10:39:18.852863 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:18.852867 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:18.852889 (+ 22us) server_negotiation.cc:407] Connection header received
0925 10:39:18.853093 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:18.853104 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:18.853166 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:18.853262 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:18.854623 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:18.855344 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:18.856535 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:18.856906 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:18.856991 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:18.860767 (+ 3776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:18.860772 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:18.860776 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:18.860809 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:18.863306 (+ 2497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:18.863875 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:18.863878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:18.863881 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:18.863995 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:18.864302 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:18.864305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:18.864307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:18.864546 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:18.864717 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:18.864934 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 10:39:18.865099 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":314,"threads_started":1}
I20260925 10:39:18.868891 25707 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43782:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:39:18.869539 25707 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:39:18.875763 25845 tablet_service.cc:1467] Processing CreateTablet for tablet 5a69e1de480445dbb87203fc21c7e54f (DEFAULT_TABLE table=test-table [id=7129575dabed4d61ac5b405293de0842]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:39:18.876000 25983 tablet_service.cc:1467] Processing CreateTablet for tablet 5a69e1de480445dbb87203fc21c7e54f (DEFAULT_TABLE table=test-table [id=7129575dabed4d61ac5b405293de0842]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:39:18.876230 25845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a69e1de480445dbb87203fc21c7e54f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:39:18.876209 26123 tablet_service.cc:1467] Processing CreateTablet for tablet 5a69e1de480445dbb87203fc21c7e54f (DEFAULT_TABLE table=test-table [id=7129575dabed4d61ac5b405293de0842]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:39:18.876441 25983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a69e1de480445dbb87203fc21c7e54f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:39:18.876585 26123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a69e1de480445dbb87203fc21c7e54f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:39:18.882994 26305 tablet_bootstrap.cc:492] T 5a69e1de480445dbb87203fc21c7e54f P 5ae87327c48c44938d940dd9995a4d7e: Bootstrap starting.
I20260925 10:39:18.883594 26307 tablet_bootstrap.cc:492] T 5a69e1de480445dbb87203fc21c7e54f P f216ed6b172f40999704a36c3b47bfdd: Bootstrap starting.
I20260925 10:39:18.884925 26305 tablet_bootstrap.cc:654] T 5a69e1de480445dbb87203fc21c7e54f P 5ae87327c48c44938d940dd9995a4d7e: Neither blocks nor log segments found. Creating new log.
I20260925 10:39:18.885406 26307 tablet_bootstrap.cc:654] T 5a69e1de480445dbb87203fc21c7e54f P f216ed6b172f40999704a36c3b47bfdd: Neither blocks nor log segments found. Creating new log.
I20260925 10:39:18.887022 26306 tablet_bootstrap.cc:492] T 5a69e1de480445dbb87203fc21c7e54f P b87295069e6d49a49c4e68fb5af6fce0: Bootstrap starting.
I20260925 10:39:18.889204 26306 tablet_bootstrap.cc:654] T 5a69e1de480445dbb87203fc21c7e54f P b87295069e6d49a49c4e68fb5af6fce0: Neither blocks nor log segments found. Creating new log.
I20260925 10:39:18.890658 26307 tablet_bootstrap.cc:492] T 5a69e1de480445dbb87203fc21c7e54f P f216ed6b172f40999704a36c3b47bfdd: No bootstrap required, opened a new log
I20260925 10:39:18.890861 26307 ts_tablet_manager.cc:1397] T 5a69e1de480445dbb87203fc21c7e54f P f216ed6b172f40999704a36c3b47bfdd: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260925 10:39:18.891451 26305 tablet_bootstrap.cc:492] T 5a69e1de480445dbb87203fc21c7e54f P 5ae87327c48c44938d940dd9995a4d7e: No bootstrap required, opened a new log
I20260925 10:39:18.891618 26305 ts_tablet_manager.cc:1397] T 5a69e1de480445dbb87203fc21c7e54f P 5ae87327c48c44938d940dd9995a4d7e: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.006s
I20260925 10:39:18.891511 26307 raft_consensus.cc:348] T 5a69e1de480445dbb87203fc21c7e54f P f216ed6b172f40999704a36c3b47bfdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } } peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } }
I20260925 10:39:18.891786 26307 raft_consensus.cc:374] T 5a69e1de480445dbb87203fc21c7e54f P f216ed6b172f40999704a36c3b47bfdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:39:18.891897 26307 raft_consensus.cc:729] T 5a69e1de480445dbb87203fc21c7e54f P f216ed6b172f40999704a36c3b47bfdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f216ed6b172f40999704a36c3b47bfdd, State: Initialized, Role: FOLLOWER
I20260925 10:39:18.892153 26307 consensus_queue.cc:260] T 5a69e1de480445dbb87203fc21c7e54f P f216ed6b172f40999704a36c3b47bfdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } } peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } }
I20260925 10:39:18.892483 26305 raft_consensus.cc:348] T 5a69e1de480445dbb87203fc21c7e54f P 5ae87327c48c44938d940dd9995a4d7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } } peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } }
I20260925 10:39:18.892745 26307 ts_tablet_manager.cc:1428] T 5a69e1de480445dbb87203fc21c7e54f P f216ed6b172f40999704a36c3b47bfdd: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260925 10:39:18.892755 26305 raft_consensus.cc:374] T 5a69e1de480445dbb87203fc21c7e54f P 5ae87327c48c44938d940dd9995a4d7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:39:18.892846 26305 raft_consensus.cc:729] T 5a69e1de480445dbb87203fc21c7e54f P 5ae87327c48c44938d940dd9995a4d7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ae87327c48c44938d940dd9995a4d7e, State: Initialized, Role: FOLLOWER
I20260925 10:39:18.893055 26305 consensus_queue.cc:260] T 5a69e1de480445dbb87203fc21c7e54f P 5ae87327c48c44938d940dd9995a4d7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } } peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } }
I20260925 10:39:18.893908 26306 tablet_bootstrap.cc:492] T 5a69e1de480445dbb87203fc21c7e54f P b87295069e6d49a49c4e68fb5af6fce0: No bootstrap required, opened a new log
I20260925 10:39:18.893894 26305 ts_tablet_manager.cc:1428] T 5a69e1de480445dbb87203fc21c7e54f P 5ae87327c48c44938d940dd9995a4d7e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 10:39:18.894098 26306 ts_tablet_manager.cc:1397] T 5a69e1de480445dbb87203fc21c7e54f P b87295069e6d49a49c4e68fb5af6fce0: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.002s
I20260925 10:39:18.895172 26306 raft_consensus.cc:348] T 5a69e1de480445dbb87203fc21c7e54f P b87295069e6d49a49c4e68fb5af6fce0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } } peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } }
I20260925 10:39:18.895546 26306 raft_consensus.cc:374] T 5a69e1de480445dbb87203fc21c7e54f P b87295069e6d49a49c4e68fb5af6fce0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:39:18.895660 26306 raft_consensus.cc:729] T 5a69e1de480445dbb87203fc21c7e54f P b87295069e6d49a49c4e68fb5af6fce0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87295069e6d49a49c4e68fb5af6fce0, State: Initialized, Role: FOLLOWER
I20260925 10:39:18.895895 26306 consensus_queue.cc:260] T 5a69e1de480445dbb87203fc21c7e54f P b87295069e6d49a49c4e68fb5af6fce0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } } peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } }
I20260925 10:39:18.896584 26306 ts_tablet_manager.cc:1428] T 5a69e1de480445dbb87203fc21c7e54f P b87295069e6d49a49c4e68fb5af6fce0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 10:39:19.150053 26311 raft_consensus.cc:482] T 5a69e1de480445dbb87203fc21c7e54f P 5ae87327c48c44938d940dd9995a4d7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:39:19.150395 26311 raft_consensus.cc:504] T 5a69e1de480445dbb87203fc21c7e54f P 5ae87327c48c44938d940dd9995a4d7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } } peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } }
I20260925 10:39:19.152103 26311 leader_election.cc:288] T 5a69e1de480445dbb87203fc21c7e54f P 5ae87327c48c44938d940dd9995a4d7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b87295069e6d49a49c4e68fb5af6fce0 (127.23.100.2:33655), f216ed6b172f40999704a36c3b47bfdd (127.23.100.3:41541)
I20260925 10:39:19.158804 26312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:19.152486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:50225 (local address 127.23.100.2:33655)
0925 10:39:19.153098 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:19.153102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:19.153350 (+ 248us) server_negotiation.cc:407] Connection header received
0925 10:39:19.153555 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:19.153560 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:19.153641 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:19.153756 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:39:19.154527 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:19.155299 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:19.156975 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:19.157800 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:19.157990 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:19.158072 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:19.158348 (+ 276us) server_negotiation.cc:299] Negotiation successful
0925 10:39:19.158477 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":326,"threads_started":1}
I20260925 10:39:19.158810 26313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:19.152466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.2:33655 (local address 127.23.100.1:50225)
0925 10:39:19.153199 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:19.153220 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:19.153365 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:19.154014 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:19.154019 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:19.154048 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:39:19.154359 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:19.154378 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:19.155438 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:19.155443 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:19.156825 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:19.156837 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:19.157995 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:19.158001 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:19.158167 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:19.158190 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 10:39:19.158300 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 10:39:19.158452 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":432,"threads_started":1}
I20260925 10:39:19.159216 26315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:19.153550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:58983 (local address 127.23.100.3:41541)
0925 10:39:19.154222 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:19.154228 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:19.154255 (+ 27us) server_negotiation.cc:407] Connection header received
0925 10:39:19.154356 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:19.154363 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:19.154421 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:19.154522 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:39:19.155081 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:19.155802 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:19.157466 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:19.158219 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:19.158428 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:19.158500 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:19.158783 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 10:39:19.158911 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":386,"threads_started":1}
I20260925 10:39:19.159245 26314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:19.153331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.3:41541 (local address 127.23.100.1:58983)
0925 10:39:19.153958 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:19.153978 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:19.154105 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:19.154550 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:19.154554 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:19.154576 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:39:19.154910 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:19.154918 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:19.155936 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:19.155944 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:19.157297 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:19.157308 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:19.158457 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:19.158464 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:19.158623 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:19.158647 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 10:39:19.158745 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 10:39:19.158870 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":299,"threads_started":1}
I20260925 10:39:19.159780 26006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a69e1de480445dbb87203fc21c7e54f" candidate_uuid: "5ae87327c48c44938d940dd9995a4d7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87295069e6d49a49c4e68fb5af6fce0" is_pre_election: true
I20260925 10:39:19.159803 26147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a69e1de480445dbb87203fc21c7e54f" candidate_uuid: "5ae87327c48c44938d940dd9995a4d7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f216ed6b172f40999704a36c3b47bfdd" is_pre_election: true
I20260925 10:39:19.160027 26147 raft_consensus.cc:2454] T 5a69e1de480445dbb87203fc21c7e54f P f216ed6b172f40999704a36c3b47bfdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ae87327c48c44938d940dd9995a4d7e in term 0.
I20260925 10:39:19.160032 26006 raft_consensus.cc:2454] T 5a69e1de480445dbb87203fc21c7e54f P b87295069e6d49a49c4e68fb5af6fce0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ae87327c48c44938d940dd9995a4d7e in term 0.
I20260925 10:39:19.160534 25798 leader_election.cc:302] T 5a69e1de480445dbb87203fc21c7e54f P 5ae87327c48c44938d940dd9995a4d7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ae87327c48c44938d940dd9995a4d7e, f216ed6b172f40999704a36c3b47bfdd; no voters:
I20260925 10:39:19.160913 26311 raft_consensus.cc:2781] T 5a69e1de480445dbb87203fc21c7e54f P 5ae87327c48c44938d940dd9995a4d7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:39:19.161049 26311 raft_consensus.cc:482] T 5a69e1de480445dbb87203fc21c7e54f P 5ae87327c48c44938d940dd9995a4d7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:39:19.161150 26311 raft_consensus.cc:3037] T 5a69e1de480445dbb87203fc21c7e54f P 5ae87327c48c44938d940dd9995a4d7e [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:39:19.163313 26311 raft_consensus.cc:504] T 5a69e1de480445dbb87203fc21c7e54f P 5ae87327c48c44938d940dd9995a4d7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } } peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } }
I20260925 10:39:19.163909 26311 leader_election.cc:288] T 5a69e1de480445dbb87203fc21c7e54f P 5ae87327c48c44938d940dd9995a4d7e [CANDIDATE]: Term 1 election: Requested vote from peers b87295069e6d49a49c4e68fb5af6fce0 (127.23.100.2:33655), f216ed6b172f40999704a36c3b47bfdd (127.23.100.3:41541)
I20260925 10:39:19.164291 26006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a69e1de480445dbb87203fc21c7e54f" candidate_uuid: "5ae87327c48c44938d940dd9995a4d7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87295069e6d49a49c4e68fb5af6fce0"
I20260925 10:39:19.164443 26147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a69e1de480445dbb87203fc21c7e54f" candidate_uuid: "5ae87327c48c44938d940dd9995a4d7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f216ed6b172f40999704a36c3b47bfdd"
I20260925 10:39:19.164574 26006 raft_consensus.cc:3037] T 5a69e1de480445dbb87203fc21c7e54f P b87295069e6d49a49c4e68fb5af6fce0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:39:19.164642 26147 raft_consensus.cc:3037] T 5a69e1de480445dbb87203fc21c7e54f P f216ed6b172f40999704a36c3b47bfdd [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:39:19.167526 26147 raft_consensus.cc:2454] T 5a69e1de480445dbb87203fc21c7e54f P f216ed6b172f40999704a36c3b47bfdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ae87327c48c44938d940dd9995a4d7e in term 1.
I20260925 10:39:19.167510 26006 raft_consensus.cc:2454] T 5a69e1de480445dbb87203fc21c7e54f P b87295069e6d49a49c4e68fb5af6fce0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ae87327c48c44938d940dd9995a4d7e in term 1.
I20260925 10:39:19.167879 25798 leader_election.cc:302] T 5a69e1de480445dbb87203fc21c7e54f P 5ae87327c48c44938d940dd9995a4d7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ae87327c48c44938d940dd9995a4d7e, f216ed6b172f40999704a36c3b47bfdd; no voters:
I20260925 10:39:19.168141 26311 raft_consensus.cc:2781] T 5a69e1de480445dbb87203fc21c7e54f P 5ae87327c48c44938d940dd9995a4d7e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:39:19.168670 26311 raft_consensus.cc:686] T 5a69e1de480445dbb87203fc21c7e54f P 5ae87327c48c44938d940dd9995a4d7e [term 1 LEADER]: Becoming Leader. State: Replica: 5ae87327c48c44938d940dd9995a4d7e, State: Running, Role: LEADER
I20260925 10:39:19.169246 26311 consensus_queue.cc:237] T 5a69e1de480445dbb87203fc21c7e54f P 5ae87327c48c44938d940dd9995a4d7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } } peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } }
I20260925 10:39:19.172614 25708 catalog_manager.cc:5495] T 5a69e1de480445dbb87203fc21c7e54f P 5ae87327c48c44938d940dd9995a4d7e reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ae87327c48c44938d940dd9995a4d7e (127.23.100.1). New cstate: current_term: 1 leader_uuid: "5ae87327c48c44938d940dd9995a4d7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } health_report { overall_health: UNKNOWN } } }
I20260925 10:39:19.257496 26319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:19.252126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33630 (local address 127.23.100.1:42361)
0925 10:39:19.252699 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:19.252703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:19.252719 (+ 16us) server_negotiation.cc:407] Connection header received
0925 10:39:19.252794 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:19.252797 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:19.252871 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:19.253072 (+ 201us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:39:19.253370 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:19.254121 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:19.255417 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:19.255839 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:19.255928 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:19.256304 (+ 376us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:39:19.256880 (+ 576us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:39:19.256987 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:19.257053 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 10:39:19.257110 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":299,"threads_started":1}
I20260925 10:39:19.270939 26312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:19.266299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:58319 (local address 127.23.100.2:33655)
0925 10:39:19.266493 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:19.266497 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:19.266962 (+ 465us) server_negotiation.cc:407] Connection header received
0925 10:39:19.267140 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:19.267144 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:19.267206 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:19.267279 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:39:19.267672 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:19.268325 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:19.269532 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:19.269907 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:19.269994 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:19.270412 (+ 418us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:39:19.270498 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:39:19.270565 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:19.270689 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 10:39:19.270745 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 10:39:19.271008 26322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:19.266192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.2:33655 (local address 127.23.100.1:58319)
0925 10:39:19.266885 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:19.266899 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:19.266998 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:19.267297 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:19.267300 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:19.267308 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:39:19.267560 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:19.267566 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:19.268427 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:19.268430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:19.269421 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:19.269431 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:19.270050 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:19.270054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:19.270178 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:19.270224 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:39:19.270580 (+ 356us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:39:19.270593 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 10:39:19.270656 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 10:39:19.270744 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":367,"threads_started":1}
I20260925 10:39:19.278707 26006 raft_consensus.cc:1261] T 5a69e1de480445dbb87203fc21c7e54f P b87295069e6d49a49c4e68fb5af6fce0 [term 1 FOLLOWER]: Refusing update from remote peer 5ae87327c48c44938d940dd9995a4d7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:39:19.278723 26147 raft_consensus.cc:1261] T 5a69e1de480445dbb87203fc21c7e54f P f216ed6b172f40999704a36c3b47bfdd [term 1 FOLLOWER]: Refusing update from remote peer 5ae87327c48c44938d940dd9995a4d7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:39:19.279356 26316 consensus_queue.cc:1035] T 5a69e1de480445dbb87203fc21c7e54f P 5ae87327c48c44938d940dd9995a4d7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:39:19.279744 26311 consensus_queue.cc:1035] T 5a69e1de480445dbb87203fc21c7e54f P 5ae87327c48c44938d940dd9995a4d7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:39:19.304615 26319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:19.299076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:37355 (local address 127.23.100.1:42361)
0925 10:39:19.299277 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:19.299283 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:19.299662 (+ 379us) server_negotiation.cc:407] Connection header received
0925 10:39:19.299859 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:19.299864 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:19.299937 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:19.300025 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:39:19.300491 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:19.301207 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:19.302909 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:19.303321 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:19.303462 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:19.303839 (+ 377us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:39:19.303938 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:39:19.304032 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:19.304191 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 10:39:19.304262 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 10:39:19.304651 26327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:19.298923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:42361 (local address 127.23.100.2:37355)
0925 10:39:19.299548 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:19.299570 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:19.299727 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:19.300058 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:19.300065 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:19.300077 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:39:19.300355 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:19.300363 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:19.301336 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:19.301344 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:19.302769 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:19.302781 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:19.303496 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:19.303503 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:19.303646 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:19.303680 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:39:19.304040 (+ 360us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:39:19.304072 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 10:39:19.304170 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 10:39:19.304299 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":271,"threads_started":1}
I20260925 10:39:19.336436 25708 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:39:19.336694 25708 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:39:19.339823 25708 catalog_manager.cc:5782] T 00000000000000000000000000000000 P beeffe7071f041e297918aa03aa556c9: Sending DeleteTablet for 3 replicas of tablet 5a69e1de480445dbb87203fc21c7e54f
I20260925 10:39:19.340561 25845 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a69e1de480445dbb87203fc21c7e54f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:39:19 UTC) from {username='kudu', principal='kudu/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:33618
I20260925 10:39:19.340644 25983 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a69e1de480445dbb87203fc21c7e54f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:39:19 UTC) from {username='kudu', principal='kudu/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:53270
I20260925 10:39:19.340909 26123 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a69e1de480445dbb87203fc21c7e54f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:39:19 UTC) from {username='kudu', principal='kudu/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:35528
I20260925 10:39:19.344347 25707 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43782:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:39:19.345010 25707 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:39:19.352965 26330 tablet_replica.cc:330] T 5a69e1de480445dbb87203fc21c7e54f P 5ae87327c48c44938d940dd9995a4d7e: stopping tablet replica
I20260925 10:39:19.353304 26330 raft_consensus.cc:2229] T 5a69e1de480445dbb87203fc21c7e54f P 5ae87327c48c44938d940dd9995a4d7e [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:39:19.353842 25984 tablet_service.cc:1467] Processing CreateTablet for tablet 61f9ee5f2f3649d09da47637d8d4af34 (DEFAULT_TABLE table=test-table [id=ae7d19e5ff6b413c9d478b446fc5708d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:39:19.354140 25845 tablet_service.cc:1467] Processing CreateTablet for tablet 61f9ee5f2f3649d09da47637d8d4af34 (DEFAULT_TABLE table=test-table [id=ae7d19e5ff6b413c9d478b446fc5708d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:39:19.354342 25984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61f9ee5f2f3649d09da47637d8d4af34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:39:19.354681 25845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61f9ee5f2f3649d09da47637d8d4af34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:39:19.358408 26331 tablet_replica.cc:330] T 5a69e1de480445dbb87203fc21c7e54f P b87295069e6d49a49c4e68fb5af6fce0: stopping tablet replica
I20260925 10:39:19.358600 26331 raft_consensus.cc:2229] T 5a69e1de480445dbb87203fc21c7e54f P b87295069e6d49a49c4e68fb5af6fce0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:39:19.358835 26331 raft_consensus.cc:2258] T 5a69e1de480445dbb87203fc21c7e54f P b87295069e6d49a49c4e68fb5af6fce0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:39:19.358832 26330 raft_consensus.cc:2258] T 5a69e1de480445dbb87203fc21c7e54f P 5ae87327c48c44938d940dd9995a4d7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:39:19.359232 26122 tablet_service.cc:1467] Processing CreateTablet for tablet 61f9ee5f2f3649d09da47637d8d4af34 (DEFAULT_TABLE table=test-table [id=ae7d19e5ff6b413c9d478b446fc5708d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:39:19.359650 26122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61f9ee5f2f3649d09da47637d8d4af34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:39:19.361660 26305 tablet_bootstrap.cc:492] T 61f9ee5f2f3649d09da47637d8d4af34 P 5ae87327c48c44938d940dd9995a4d7e: Bootstrap starting.
I20260925 10:39:19.363407 26305 tablet_bootstrap.cc:654] T 61f9ee5f2f3649d09da47637d8d4af34 P 5ae87327c48c44938d940dd9995a4d7e: Neither blocks nor log segments found. Creating new log.
I20260925 10:39:19.366493 26330 ts_tablet_manager.cc:1905] T 5a69e1de480445dbb87203fc21c7e54f P 5ae87327c48c44938d940dd9995a4d7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:39:19.366645 26307 tablet_bootstrap.cc:492] T 61f9ee5f2f3649d09da47637d8d4af34 P f216ed6b172f40999704a36c3b47bfdd: Bootstrap starting.
I20260925 10:39:19.368335 26307 tablet_bootstrap.cc:654] T 61f9ee5f2f3649d09da47637d8d4af34 P f216ed6b172f40999704a36c3b47bfdd: Neither blocks nor log segments found. Creating new log.
I20260925 10:39:19.370817 26306 tablet_bootstrap.cc:492] T 61f9ee5f2f3649d09da47637d8d4af34 P b87295069e6d49a49c4e68fb5af6fce0: Bootstrap starting.
I20260925 10:39:19.372726 26306 tablet_bootstrap.cc:654] T 61f9ee5f2f3649d09da47637d8d4af34 P b87295069e6d49a49c4e68fb5af6fce0: Neither blocks nor log segments found. Creating new log.
I20260925 10:39:19.373898 26331 ts_tablet_manager.cc:1905] T 5a69e1de480445dbb87203fc21c7e54f P b87295069e6d49a49c4e68fb5af6fce0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:39:19.379597 26330 ts_tablet_manager.cc:1918] T 5a69e1de480445dbb87203fc21c7e54f P 5ae87327c48c44938d940dd9995a4d7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:39:19.379758 26330 log.cc:1196] T 5a69e1de480445dbb87203fc21c7e54f P 5ae87327c48c44938d940dd9995a4d7e: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-0/wal/wals/5a69e1de480445dbb87203fc21c7e54f
I20260925 10:39:19.380280 26330 ts_tablet_manager.cc:1939] T 5a69e1de480445dbb87203fc21c7e54f P 5ae87327c48c44938d940dd9995a4d7e: Deleting consensus metadata
I20260925 10:39:19.381017 26305 tablet_bootstrap.cc:492] T 61f9ee5f2f3649d09da47637d8d4af34 P 5ae87327c48c44938d940dd9995a4d7e: No bootstrap required, opened a new log
I20260925 10:39:19.381294 26307 tablet_bootstrap.cc:492] T 61f9ee5f2f3649d09da47637d8d4af34 P f216ed6b172f40999704a36c3b47bfdd: No bootstrap required, opened a new log
I20260925 10:39:19.381518 26307 ts_tablet_manager.cc:1397] T 61f9ee5f2f3649d09da47637d8d4af34 P f216ed6b172f40999704a36c3b47bfdd: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.007s
I20260925 10:39:19.382961 26332 tablet_replica.cc:330] T 5a69e1de480445dbb87203fc21c7e54f P f216ed6b172f40999704a36c3b47bfdd: stopping tablet replica
I20260925 10:39:19.383152 26332 raft_consensus.cc:2229] T 5a69e1de480445dbb87203fc21c7e54f P f216ed6b172f40999704a36c3b47bfdd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:39:19.383358 26332 raft_consensus.cc:2258] T 5a69e1de480445dbb87203fc21c7e54f P f216ed6b172f40999704a36c3b47bfdd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:39:19.383635 26305 ts_tablet_manager.cc:1397] T 61f9ee5f2f3649d09da47637d8d4af34 P 5ae87327c48c44938d940dd9995a4d7e: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.009s
I20260925 10:39:19.383947 26331 ts_tablet_manager.cc:1918] T 5a69e1de480445dbb87203fc21c7e54f P b87295069e6d49a49c4e68fb5af6fce0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:39:19.384116 26331 log.cc:1196] T 5a69e1de480445dbb87203fc21c7e54f P b87295069e6d49a49c4e68fb5af6fce0: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-1/wal/wals/5a69e1de480445dbb87203fc21c7e54f
I20260925 10:39:19.384059 26307 raft_consensus.cc:348] T 61f9ee5f2f3649d09da47637d8d4af34 P f216ed6b172f40999704a36c3b47bfdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } } peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } }
I20260925 10:39:19.384325 26307 raft_consensus.cc:374] T 61f9ee5f2f3649d09da47637d8d4af34 P f216ed6b172f40999704a36c3b47bfdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:39:19.384406 26307 raft_consensus.cc:729] T 61f9ee5f2f3649d09da47637d8d4af34 P f216ed6b172f40999704a36c3b47bfdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f216ed6b172f40999704a36c3b47bfdd, State: Initialized, Role: FOLLOWER
I20260925 10:39:19.384490 26305 raft_consensus.cc:348] T 61f9ee5f2f3649d09da47637d8d4af34 P 5ae87327c48c44938d940dd9995a4d7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } } peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } }
I20260925 10:39:19.384758 26305 raft_consensus.cc:374] T 61f9ee5f2f3649d09da47637d8d4af34 P 5ae87327c48c44938d940dd9995a4d7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:39:19.384690 26307 consensus_queue.cc:260] T 61f9ee5f2f3649d09da47637d8d4af34 P f216ed6b172f40999704a36c3b47bfdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } } peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } }
I20260925 10:39:19.384850 26305 raft_consensus.cc:729] T 61f9ee5f2f3649d09da47637d8d4af34 P 5ae87327c48c44938d940dd9995a4d7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ae87327c48c44938d940dd9995a4d7e, State: Initialized, Role: FOLLOWER
I20260925 10:39:19.384951 26331 ts_tablet_manager.cc:1939] T 5a69e1de480445dbb87203fc21c7e54f P b87295069e6d49a49c4e68fb5af6fce0: Deleting consensus metadata
I20260925 10:39:19.385126 26305 consensus_queue.cc:260] T 61f9ee5f2f3649d09da47637d8d4af34 P 5ae87327c48c44938d940dd9995a4d7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } } peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } }
I20260925 10:39:19.385645 26305 ts_tablet_manager.cc:1428] T 61f9ee5f2f3649d09da47637d8d4af34 P 5ae87327c48c44938d940dd9995a4d7e: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260925 10:39:19.386596 26307 ts_tablet_manager.cc:1428] T 61f9ee5f2f3649d09da47637d8d4af34 P f216ed6b172f40999704a36c3b47bfdd: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20260925 10:39:19.386775 25696 catalog_manager.cc:4841] TS 5ae87327c48c44938d940dd9995a4d7e (127.23.100.1:42361): tablet 5a69e1de480445dbb87203fc21c7e54f (table test-table [id=7129575dabed4d61ac5b405293de0842]) successfully deleted
I20260925 10:39:19.387830 26332 ts_tablet_manager.cc:1905] T 5a69e1de480445dbb87203fc21c7e54f P f216ed6b172f40999704a36c3b47bfdd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:39:19.387939 25695 catalog_manager.cc:4841] TS b87295069e6d49a49c4e68fb5af6fce0 (127.23.100.2:33655): tablet 5a69e1de480445dbb87203fc21c7e54f (table test-table [id=7129575dabed4d61ac5b405293de0842]) successfully deleted
I20260925 10:39:19.391160 26306 tablet_bootstrap.cc:492] T 61f9ee5f2f3649d09da47637d8d4af34 P b87295069e6d49a49c4e68fb5af6fce0: No bootstrap required, opened a new log
I20260925 10:39:19.391330 26306 ts_tablet_manager.cc:1397] T 61f9ee5f2f3649d09da47637d8d4af34 P b87295069e6d49a49c4e68fb5af6fce0: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.008s
I20260925 10:39:19.392056 26306 raft_consensus.cc:348] T 61f9ee5f2f3649d09da47637d8d4af34 P b87295069e6d49a49c4e68fb5af6fce0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } } peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } }
I20260925 10:39:19.392326 26306 raft_consensus.cc:374] T 61f9ee5f2f3649d09da47637d8d4af34 P b87295069e6d49a49c4e68fb5af6fce0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:39:19.392416 26306 raft_consensus.cc:729] T 61f9ee5f2f3649d09da47637d8d4af34 P b87295069e6d49a49c4e68fb5af6fce0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87295069e6d49a49c4e68fb5af6fce0, State: Initialized, Role: FOLLOWER
I20260925 10:39:19.392668 26306 consensus_queue.cc:260] T 61f9ee5f2f3649d09da47637d8d4af34 P b87295069e6d49a49c4e68fb5af6fce0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } } peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } }
I20260925 10:39:19.393318 26306 ts_tablet_manager.cc:1428] T 61f9ee5f2f3649d09da47637d8d4af34 P b87295069e6d49a49c4e68fb5af6fce0: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260925 10:39:19.393862 26332 ts_tablet_manager.cc:1918] T 5a69e1de480445dbb87203fc21c7e54f P f216ed6b172f40999704a36c3b47bfdd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:39:19.394014 26332 log.cc:1196] T 5a69e1de480445dbb87203fc21c7e54f P f216ed6b172f40999704a36c3b47bfdd: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-2/wal/wals/5a69e1de480445dbb87203fc21c7e54f
I20260925 10:39:19.394413 26332 ts_tablet_manager.cc:1939] T 5a69e1de480445dbb87203fc21c7e54f P f216ed6b172f40999704a36c3b47bfdd: Deleting consensus metadata
I20260925 10:39:19.395709 25695 catalog_manager.cc:4841] TS f216ed6b172f40999704a36c3b47bfdd (127.23.100.3:41541): tablet 5a69e1de480445dbb87203fc21c7e54f (table test-table [id=7129575dabed4d61ac5b405293de0842]) successfully deleted
I20260925 10:39:19.405966 26275 raft_consensus.cc:482] T 61f9ee5f2f3649d09da47637d8d4af34 P b87295069e6d49a49c4e68fb5af6fce0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:39:19.406129 26275 raft_consensus.cc:504] T 61f9ee5f2f3649d09da47637d8d4af34 P b87295069e6d49a49c4e68fb5af6fce0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } } peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } }
I20260925 10:39:19.406886 26275 leader_election.cc:288] T 61f9ee5f2f3649d09da47637d8d4af34 P b87295069e6d49a49c4e68fb5af6fce0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ae87327c48c44938d940dd9995a4d7e (127.23.100.1:42361), f216ed6b172f40999704a36c3b47bfdd (127.23.100.3:41541)
I20260925 10:39:19.407351 26147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f9ee5f2f3649d09da47637d8d4af34" candidate_uuid: "b87295069e6d49a49c4e68fb5af6fce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f216ed6b172f40999704a36c3b47bfdd" is_pre_election: true
I20260925 10:39:19.407361 25870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f9ee5f2f3649d09da47637d8d4af34" candidate_uuid: "b87295069e6d49a49c4e68fb5af6fce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ae87327c48c44938d940dd9995a4d7e" is_pre_election: true
I20260925 10:39:19.407560 26147 raft_consensus.cc:2454] T 61f9ee5f2f3649d09da47637d8d4af34 P f216ed6b172f40999704a36c3b47bfdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87295069e6d49a49c4e68fb5af6fce0 in term 0.
I20260925 10:39:19.407621 25870 raft_consensus.cc:2454] T 61f9ee5f2f3649d09da47637d8d4af34 P 5ae87327c48c44938d940dd9995a4d7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87295069e6d49a49c4e68fb5af6fce0 in term 0.
I20260925 10:39:19.407976 25937 leader_election.cc:302] T 61f9ee5f2f3649d09da47637d8d4af34 P b87295069e6d49a49c4e68fb5af6fce0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b87295069e6d49a49c4e68fb5af6fce0, f216ed6b172f40999704a36c3b47bfdd; no voters:
I20260925 10:39:19.408241 26275 raft_consensus.cc:2781] T 61f9ee5f2f3649d09da47637d8d4af34 P b87295069e6d49a49c4e68fb5af6fce0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:39:19.408378 26275 raft_consensus.cc:482] T 61f9ee5f2f3649d09da47637d8d4af34 P b87295069e6d49a49c4e68fb5af6fce0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:39:19.408468 26275 raft_consensus.cc:3037] T 61f9ee5f2f3649d09da47637d8d4af34 P b87295069e6d49a49c4e68fb5af6fce0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:39:19.411412 26275 raft_consensus.cc:504] T 61f9ee5f2f3649d09da47637d8d4af34 P b87295069e6d49a49c4e68fb5af6fce0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } } peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } }
I20260925 10:39:19.411993 26275 leader_election.cc:288] T 61f9ee5f2f3649d09da47637d8d4af34 P b87295069e6d49a49c4e68fb5af6fce0 [CANDIDATE]: Term 1 election: Requested vote from peers 5ae87327c48c44938d940dd9995a4d7e (127.23.100.1:42361), f216ed6b172f40999704a36c3b47bfdd (127.23.100.3:41541)
I20260925 10:39:19.412479 26147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f9ee5f2f3649d09da47637d8d4af34" candidate_uuid: "b87295069e6d49a49c4e68fb5af6fce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f216ed6b172f40999704a36c3b47bfdd"
I20260925 10:39:19.412699 26147 raft_consensus.cc:3037] T 61f9ee5f2f3649d09da47637d8d4af34 P f216ed6b172f40999704a36c3b47bfdd [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:39:19.414974 25870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f9ee5f2f3649d09da47637d8d4af34" candidate_uuid: "b87295069e6d49a49c4e68fb5af6fce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ae87327c48c44938d940dd9995a4d7e"
I20260925 10:39:19.415230 25870 raft_consensus.cc:3037] T 61f9ee5f2f3649d09da47637d8d4af34 P 5ae87327c48c44938d940dd9995a4d7e [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:39:19.415488 26147 raft_consensus.cc:2454] T 61f9ee5f2f3649d09da47637d8d4af34 P f216ed6b172f40999704a36c3b47bfdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87295069e6d49a49c4e68fb5af6fce0 in term 1.
I20260925 10:39:19.415863 25937 leader_election.cc:302] T 61f9ee5f2f3649d09da47637d8d4af34 P b87295069e6d49a49c4e68fb5af6fce0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b87295069e6d49a49c4e68fb5af6fce0, f216ed6b172f40999704a36c3b47bfdd; no voters:
I20260925 10:39:19.416127 26275 raft_consensus.cc:2781] T 61f9ee5f2f3649d09da47637d8d4af34 P b87295069e6d49a49c4e68fb5af6fce0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:39:19.416265 26275 raft_consensus.cc:686] T 61f9ee5f2f3649d09da47637d8d4af34 P b87295069e6d49a49c4e68fb5af6fce0 [term 1 LEADER]: Becoming Leader. State: Replica: b87295069e6d49a49c4e68fb5af6fce0, State: Running, Role: LEADER
I20260925 10:39:19.416585 26275 consensus_queue.cc:237] T 61f9ee5f2f3649d09da47637d8d4af34 P b87295069e6d49a49c4e68fb5af6fce0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } } peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } }
I20260925 10:39:19.417768 25870 raft_consensus.cc:2454] T 61f9ee5f2f3649d09da47637d8d4af34 P 5ae87327c48c44938d940dd9995a4d7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87295069e6d49a49c4e68fb5af6fce0 in term 1.
I20260925 10:39:19.419924 25708 catalog_manager.cc:5495] T 61f9ee5f2f3649d09da47637d8d4af34 P b87295069e6d49a49c4e68fb5af6fce0 reported cstate change: term changed from 0 to 1, leader changed from <none> to b87295069e6d49a49c4e68fb5af6fce0 (127.23.100.2). New cstate: current_term: 1 leader_uuid: "b87295069e6d49a49c4e68fb5af6fce0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b87295069e6d49a49c4e68fb5af6fce0" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 33655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 } health_report { overall_health: UNKNOWN } } }
I20260925 10:39:19.445053 26312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:19.439723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53302 (local address 127.23.100.2:33655)
0925 10:39:19.439897 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:19.439903 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:19.440174 (+ 271us) server_negotiation.cc:407] Connection header received
0925 10:39:19.440374 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:19.440378 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:19.440436 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:19.440500 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:39:19.440898 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:19.441583 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:19.443463 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:19.443833 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:19.443918 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:19.444406 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:39:19.444499 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:39:19.444567 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:19.444719 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 10:39:19.444782 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 10:39:19.453446 26146 raft_consensus.cc:1261] T 61f9ee5f2f3649d09da47637d8d4af34 P f216ed6b172f40999704a36c3b47bfdd [term 1 FOLLOWER]: Refusing update from remote peer b87295069e6d49a49c4e68fb5af6fce0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:39:19.453933 26275 consensus_queue.cc:1035] T 61f9ee5f2f3649d09da47637d8d4af34 P b87295069e6d49a49c4e68fb5af6fce0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f216ed6b172f40999704a36c3b47bfdd" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 41541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:39:19.454043 25870 raft_consensus.cc:1261] T 61f9ee5f2f3649d09da47637d8d4af34 P 5ae87327c48c44938d940dd9995a4d7e [term 1 FOLLOWER]: Refusing update from remote peer b87295069e6d49a49c4e68fb5af6fce0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:39:19.454514 26275 consensus_queue.cc:1035] T 61f9ee5f2f3649d09da47637d8d4af34 P b87295069e6d49a49c4e68fb5af6fce0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ae87327c48c44938d940dd9995a4d7e" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 42361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 10:39:19.473960 26273 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 10:39:21.055078 25708 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:39:21.055318 25708 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:39:21.059455 25708 catalog_manager.cc:5782] T 00000000000000000000000000000000 P beeffe7071f041e297918aa03aa556c9: Sending DeleteTablet for 3 replicas of tablet 61f9ee5f2f3649d09da47637d8d4af34
I20260925 10:39:21.060179 25845 tablet_service.cc:1514] Processing DeleteTablet for tablet 61f9ee5f2f3649d09da47637d8d4af34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:39:21 UTC) from {username='kudu', principal='kudu/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:33618
I20260925 10:39:21.060189 25983 tablet_service.cc:1514] Processing DeleteTablet for tablet 61f9ee5f2f3649d09da47637d8d4af34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:39:21 UTC) from {username='kudu', principal='kudu/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:53270
I20260925 10:39:21.060563 26123 tablet_service.cc:1514] Processing DeleteTablet for tablet 61f9ee5f2f3649d09da47637d8d4af34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:39:21 UTC) from {username='kudu', principal='kudu/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:35528
I20260925 10:39:21.061013 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 25780
I20260925 10:39:21.061256 26361 tablet_replica.cc:330] T 61f9ee5f2f3649d09da47637d8d4af34 P b87295069e6d49a49c4e68fb5af6fce0: stopping tablet replica
I20260925 10:39:21.061493 26361 raft_consensus.cc:2229] T 61f9ee5f2f3649d09da47637d8d4af34 P b87295069e6d49a49c4e68fb5af6fce0 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:39:21.061826 26361 raft_consensus.cc:2258] T 61f9ee5f2f3649d09da47637d8d4af34 P b87295069e6d49a49c4e68fb5af6fce0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:39:21.063099 26361 ts_tablet_manager.cc:1905] T 61f9ee5f2f3649d09da47637d8d4af34 P b87295069e6d49a49c4e68fb5af6fce0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:39:21.071166 26361 ts_tablet_manager.cc:1918] T 61f9ee5f2f3649d09da47637d8d4af34 P b87295069e6d49a49c4e68fb5af6fce0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 10:39:21.071321 26361 log.cc:1196] T 61f9ee5f2f3649d09da47637d8d4af34 P b87295069e6d49a49c4e68fb5af6fce0: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-1/wal/wals/61f9ee5f2f3649d09da47637d8d4af34
I20260925 10:39:21.071832 26361 ts_tablet_manager.cc:1939] T 61f9ee5f2f3649d09da47637d8d4af34 P b87295069e6d49a49c4e68fb5af6fce0: Deleting consensus metadata
I20260925 10:39:21.073002 26363 tablet_replica.cc:330] T 61f9ee5f2f3649d09da47637d8d4af34 P f216ed6b172f40999704a36c3b47bfdd: stopping tablet replica
I20260925 10:39:21.072995 25695 catalog_manager.cc:4841] TS b87295069e6d49a49c4e68fb5af6fce0 (127.23.100.2:33655): tablet 61f9ee5f2f3649d09da47637d8d4af34 (table test-table [id=ae7d19e5ff6b413c9d478b446fc5708d]) successfully deleted
I20260925 10:39:21.073199 26363 raft_consensus.cc:2229] T 61f9ee5f2f3649d09da47637d8d4af34 P f216ed6b172f40999704a36c3b47bfdd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:39:21.073302 26362 tablet_replica.cc:330] T 61f9ee5f2f3649d09da47637d8d4af34 P 5ae87327c48c44938d940dd9995a4d7e: stopping tablet replica
I20260925 10:39:21.073443 26363 raft_consensus.cc:2258] T 61f9ee5f2f3649d09da47637d8d4af34 P f216ed6b172f40999704a36c3b47bfdd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:39:21.073585 26362 raft_consensus.cc:2229] T 61f9ee5f2f3649d09da47637d8d4af34 P 5ae87327c48c44938d940dd9995a4d7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:39:21.073836 26362 raft_consensus.cc:2258] T 61f9ee5f2f3649d09da47637d8d4af34 P 5ae87327c48c44938d940dd9995a4d7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:39:21.074790 26362 ts_tablet_manager.cc:1905] T 61f9ee5f2f3649d09da47637d8d4af34 P 5ae87327c48c44938d940dd9995a4d7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:39:21.081310 26362 ts_tablet_manager.cc:1918] T 61f9ee5f2f3649d09da47637d8d4af34 P 5ae87327c48c44938d940dd9995a4d7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 10:39:21.081416 26362 log.cc:1196] T 61f9ee5f2f3649d09da47637d8d4af34 P 5ae87327c48c44938d940dd9995a4d7e: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-0/wal/wals/61f9ee5f2f3649d09da47637d8d4af34
I20260925 10:39:21.081794 26362 ts_tablet_manager.cc:1939] T 61f9ee5f2f3649d09da47637d8d4af34 P 5ae87327c48c44938d940dd9995a4d7e: Deleting consensus metadata
I20260925 10:39:21.082806 25696 catalog_manager.cc:4841] TS 5ae87327c48c44938d940dd9995a4d7e (127.23.100.1:42361): tablet 61f9ee5f2f3649d09da47637d8d4af34 (table test-table [id=ae7d19e5ff6b413c9d478b446fc5708d]) successfully deleted
I20260925 10:39:21.086938 26363 ts_tablet_manager.cc:1905] T 61f9ee5f2f3649d09da47637d8d4af34 P f216ed6b172f40999704a36c3b47bfdd: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 25 10:39:21 dist-test-slave-dxdd krb5kdc[25664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332758, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.100.1@KRBTEST.COM
I20260925 10:39:21.093499 26363 ts_tablet_manager.cc:1918] T 61f9ee5f2f3649d09da47637d8d4af34 P f216ed6b172f40999704a36c3b47bfdd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 10:39:21.093720 26363 log.cc:1196] T 61f9ee5f2f3649d09da47637d8d4af34 P f216ed6b172f40999704a36c3b47bfdd: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790332740062128-23952-0/minicluster-data/ts-2/wal/wals/61f9ee5f2f3649d09da47637d8d4af34
I20260925 10:39:21.094162 26363 ts_tablet_manager.cc:1939] T 61f9ee5f2f3649d09da47637d8d4af34 P f216ed6b172f40999704a36c3b47bfdd: Deleting consensus metadata
I20260925 10:39:21.095211 25695 catalog_manager.cc:4841] TS f216ed6b172f40999704a36c3b47bfdd (127.23.100.3:41541): tablet 61f9ee5f2f3649d09da47637d8d4af34 (table test-table [id=ae7d19e5ff6b413c9d478b446fc5708d]) successfully deleted
I20260925 10:39:21.096629 26365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:21.062347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33632 (local address 127.23.100.1:42361)
0925 10:39:21.063868 (+ 1521us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:21.063876 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:21.074828 (+ 10952us) server_negotiation.cc:407] Connection header received
0925 10:39:21.075052 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:21.075058 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:21.075137 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:21.075248 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:21.076578 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:21.077506 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:21.078845 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:21.079288 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:21.079422 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:21.089109 (+ 9687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:21.089129 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:21.089150 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:21.089190 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:21.093533 (+ 4343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:21.094345 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:21.094356 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:21.094361 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:21.094502 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:21.094919 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:21.094923 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:21.094925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:21.095362 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:21.095686 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:21.096000 (+ 314us) server_negotiation.cc:299] Negotiation successful
0925 10:39:21.096226 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1329,"thread_start_us":1192,"threads_started":1}
W20260925 10:39:21.097301 25910 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33632
W20260925 10:39:21.097906 23952 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 25780: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:39:21.098078 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 25780
I20260925 10:39:21.120435 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 25919
Sep 25 10:39:21 dist-test-slave-dxdd krb5kdc[25664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332758, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.100.2@KRBTEST.COM
I20260925 10:39:21.134648 26366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:21.121090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53308 (local address 127.23.100.2:33655)
0925 10:39:21.121584 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:21.121589 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:21.121604 (+ 15us) server_negotiation.cc:407] Connection header received
0925 10:39:21.121658 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:21.121662 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:21.121716 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:21.121807 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:21.123092 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:21.123813 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:21.124929 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:21.125312 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:21.125410 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:21.129684 (+ 4274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:21.129695 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:21.129709 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:21.129743 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:21.132382 (+ 2639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:21.132947 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:21.132954 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:21.132956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:21.133070 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:21.133356 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:21.133359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:21.133361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:21.133697 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:21.133917 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:21.134113 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 10:39:21.134265 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":267,"threads_started":1}
W20260925 10:39:21.135253 26049 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53308
W20260925 10:39:21.135810 23952 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 25919: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:39:21.135947 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 25919
I20260925 10:39:21.161330 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 26058
Sep 25 10:39:21 dist-test-slave-dxdd krb5kdc[25664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332758, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.100.3@KRBTEST.COM
I20260925 10:39:21.176216 26367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:21.162080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35554 (local address 127.23.100.3:41541)
0925 10:39:21.162574 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:21.162579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:21.162594 (+ 15us) server_negotiation.cc:407] Connection header received
0925 10:39:21.162646 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:21.162650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:21.162709 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:21.162795 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:21.164087 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:21.164790 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:21.165987 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:21.166535 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:21.166627 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:21.170952 (+ 4325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:21.170968 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:21.170983 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:21.171013 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:21.173841 (+ 2828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:21.174468 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:21.174475 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:21.174478 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:21.174606 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:21.174923 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:21.174926 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:21.174928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:21.175236 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:21.175442 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:21.175662 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 10:39:21.175833 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":270,"threads_started":1}
W20260925 10:39:21.176870 26188 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35554
W20260925 10:39:21.177418 23952 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 26058: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:39:21.177589 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 26058
I20260925 10:39:21.199988 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 25677
I20260925 10:39:21.201164 25749 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:39:21.314596 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 25677
2026-09-25T10:39:21Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8494 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:39:21 dist-test-slave-dxdd krb5kdc[26371](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:39:21 dist-test-slave-dxdd krb5kdc[26371](info): set up 2 sockets
Sep 25 10:39:21 dist-test-slave-dxdd krb5kdc[26371](info): commencing operation
krb5kdc: starting...
W20260925 10:39:22.382606 23952 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.009s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Sep 25 10:39:22 dist-test-slave-dxdd krb5kdc[26371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:39:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:39:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
I20260925 10:39:22.565361 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:34367
--webserver_interface=127.23.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:44265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.62:34367
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:39:22.876565 26384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:39:22.876874 26384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:39:22.877041 26384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:39:22.883901 26384 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:39:22.884035 26384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:39:22.884142 26384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:39:22.884238 26384 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:39:22.884316 26384 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:39:22.893622 26384 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:44265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.62:34367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:34367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:39:22.894879 26384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:39:22.896090 26384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:39:22.907423 26390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:22.907387 26392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:22.907658 26384 server_base.cc:1034] running on GCE node
W20260925 10:39:22.907459 26389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:22.908641 26384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:39:22.909929 26384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:39:22.911156 26384 hybrid_clock.cc:648] HybridClock initialized: now 1790332762911100 us; error 34 us; skew 500 ppm
I20260925 10:39:22.911475 26384 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:39:22 dist-test-slave-dxdd krb5kdc[26371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332762, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:39:22.914184 26384 init.cc:377] Logged in from keytab as kudu/127.23.100.62@KRBTEST.COM (short username kudu)
I20260925 10:39:22.915531 26384 webserver.cc:460] Webserver started at http://127.23.100.62:41907/ using document root <none> and password file <none>
I20260925 10:39:22.916172 26384 fs_manager.cc:359] Metadata directory not provided
I20260925 10:39:22.916303 26384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:39:22.916612 26384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:39:22.919406 26384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/master-0/data/instance:
uuid: "4ff51bed5c0e4f7e9f11377cbf7d2c84"
format_stamp: "Formatted at 2026-09-25 10:39:22 on dist-test-slave-dxdd"
server_key: "2f9d031a9377fa4a9e5898e60f768fde"
server_key_iv: "c64951d635612a762602c95f36882feb"
server_key_version: "encryptionkey@0"
I20260925 10:39:22.920198 26384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/master-0/wal/instance:
uuid: "4ff51bed5c0e4f7e9f11377cbf7d2c84"
format_stamp: "Formatted at 2026-09-25 10:39:22 on dist-test-slave-dxdd"
server_key: "2f9d031a9377fa4a9e5898e60f768fde"
server_key_iv: "c64951d635612a762602c95f36882feb"
server_key_version: "encryptionkey@0"
I20260925 10:39:22.925024 26384 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260925 10:39:22.928231 26399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:22.929481 26384 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:39:22.929663 26384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/master-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/master-0/wal
uuid: "4ff51bed5c0e4f7e9f11377cbf7d2c84"
format_stamp: "Formatted at 2026-09-25 10:39:22 on dist-test-slave-dxdd"
server_key: "2f9d031a9377fa4a9e5898e60f768fde"
server_key_iv: "c64951d635612a762602c95f36882feb"
server_key_version: "encryptionkey@0"
I20260925 10:39:22.929860 26384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:39:22.960439 26384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:39:22.964253 26384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:39:22.964524 26384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:39:22.982165 26384 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.62:34367
I20260925 10:39:22.982174 26450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.62:34367 every 8 connection(s)
I20260925 10:39:22.983467 26384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
I20260925 10:39:22.986974 26451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:39:22.992416 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 26384
I20260925 10:39:22.992633 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/master-0/wal/instance
I20260925 10:39:22.993222 23952 external_mini_cluster.cc:1428] Setting key 05b72930b95dd060b472b2cc255ca5f4
I20260925 10:39:22.997838 26451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ff51bed5c0e4f7e9f11377cbf7d2c84: Bootstrap starting.
I20260925 10:39:23.001528 26451 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ff51bed5c0e4f7e9f11377cbf7d2c84: Neither blocks nor log segments found. Creating new log.
I20260925 10:39:23.002771 26451 log.cc:824] T 00000000000000000000000000000000 P 4ff51bed5c0e4f7e9f11377cbf7d2c84: Log is configured to *not* fsync() on all Append() calls
I20260925 10:39:23.005777 26451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ff51bed5c0e4f7e9f11377cbf7d2c84: No bootstrap required, opened a new log
Sep 25 10:39:23 dist-test-slave-dxdd krb5kdc[26371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:39:23.015486 26451 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ff51bed5c0e4f7e9f11377cbf7d2c84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ff51bed5c0e4f7e9f11377cbf7d2c84" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 34367 } }
I20260925 10:39:23.015921 26451 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ff51bed5c0e4f7e9f11377cbf7d2c84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:39:23.016060 26451 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ff51bed5c0e4f7e9f11377cbf7d2c84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ff51bed5c0e4f7e9f11377cbf7d2c84, State: Initialized, Role: FOLLOWER
I20260925 10:39:23.016752 26451 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ff51bed5c0e4f7e9f11377cbf7d2c84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ff51bed5c0e4f7e9f11377cbf7d2c84" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 34367 } }
I20260925 10:39:23.017038 26451 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ff51bed5c0e4f7e9f11377cbf7d2c84 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:39:23.017187 26451 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ff51bed5c0e4f7e9f11377cbf7d2c84 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:39:23.017373 26451 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ff51bed5c0e4f7e9f11377cbf7d2c84 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:39:23.017745 26454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:22.995227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48998 (local address 127.23.100.62:34367)
0925 10:39:22.995990 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:22.996005 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:22.996050 (+ 45us) server_negotiation.cc:407] Connection header received
0925 10:39:22.998437 (+ 2387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:22.998464 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:22.998918 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:22.999327 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:23.000581 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:23.002612 (+ 2031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:23.004235 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:23.005224 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:23.005400 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:23.009409 (+ 4009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:23.009418 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:23.009435 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:23.009475 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:23.013424 (+ 3949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:23.014054 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:23.014062 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:23.014072 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:23.014271 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:23.014654 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:23.014658 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:23.014661 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:23.015306 (+ 645us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:23.015587 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:23.015922 (+ 335us) server_negotiation.cc:299] Negotiation successful
0925 10:39:23.016323 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":323,"threads_started":1}
I20260925 10:39:23.019943 26451 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ff51bed5c0e4f7e9f11377cbf7d2c84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ff51bed5c0e4f7e9f11377cbf7d2c84" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 34367 } }
I20260925 10:39:23.020491 26451 leader_election.cc:302] T 00000000000000000000000000000000 P 4ff51bed5c0e4f7e9f11377cbf7d2c84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ff51bed5c0e4f7e9f11377cbf7d2c84; no voters:
I20260925 10:39:23.021056 26451 leader_election.cc:288] T 00000000000000000000000000000000 P 4ff51bed5c0e4f7e9f11377cbf7d2c84 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:39:23.021205 26456 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ff51bed5c0e4f7e9f11377cbf7d2c84 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:39:23.021950 26456 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ff51bed5c0e4f7e9f11377cbf7d2c84 [term 1 LEADER]: Becoming Leader. State: Replica: 4ff51bed5c0e4f7e9f11377cbf7d2c84, State: Running, Role: LEADER
I20260925 10:39:23.022522 26456 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ff51bed5c0e4f7e9f11377cbf7d2c84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ff51bed5c0e4f7e9f11377cbf7d2c84" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 34367 } }
I20260925 10:39:23.023119 26451 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ff51bed5c0e4f7e9f11377cbf7d2c84 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:39:23.025210 26457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ff51bed5c0e4f7e9f11377cbf7d2c84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ff51bed5c0e4f7e9f11377cbf7d2c84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ff51bed5c0e4f7e9f11377cbf7d2c84" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 34367 } } }
I20260925 10:39:23.025640 26457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ff51bed5c0e4f7e9f11377cbf7d2c84 [sys.catalog]: This master's current role is: LEADER
I20260925 10:39:23.025513 26458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ff51bed5c0e4f7e9f11377cbf7d2c84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ff51bed5c0e4f7e9f11377cbf7d2c84. Latest consensus state: current_term: 1 leader_uuid: "4ff51bed5c0e4f7e9f11377cbf7d2c84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ff51bed5c0e4f7e9f11377cbf7d2c84" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 34367 } } }
I20260925 10:39:23.025802 26458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ff51bed5c0e4f7e9f11377cbf7d2c84 [sys.catalog]: This master's current role is: LEADER
I20260925 10:39:23.029287 26464 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:39:23.034085 26464 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:39:23.043803 26464 catalog_manager.cc:1347] Generated new cluster ID: 680b2886ac17401fabe28121a80eb5de
I20260925 10:39:23.043928 26464 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:39:23.067811 26464 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:39:23.069123 26464 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:39:23.076380 26464 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ff51bed5c0e4f7e9f11377cbf7d2c84: Generated new TSK 0
I20260925 10:39:23.077148 26464 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab.
Entry for principal kudu/127.23.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab.
I20260925 10:39:23.133612 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.1:0
--local_ip_for_outbound_sockets=127.23.100.1
--webserver_interface=127.23.100.1
--webserver_port=0
--tserver_master_addrs=127.23.100.62:34367
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab
--principal=kudu/127.23.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:44265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:39:23.449698 26477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:39:23.450006 26477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:39:23.450162 26477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:39:23.457123 26477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:39:23.457252 26477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:39:23.457445 26477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.1
I20260925 10:39:23.466391 26477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:44265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab
--principal=kudu/127.23.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.1
--webserver_port=0
--tserver_master_addrs=127.23.100.62:34367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.1
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:39:23.467656 26477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:39:23.468886 26477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:39:23.481470 26483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:23.481554 26482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:23.481529 26485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:23.482635 26477 server_base.cc:1034] running on GCE node
I20260925 10:39:23.483223 26477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:39:23.484015 26477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:39:23.485203 26477 hybrid_clock.cc:648] HybridClock initialized: now 1790332763485143 us; error 54 us; skew 500 ppm
I20260925 10:39:23.485450 26477 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:39:23 dist-test-slave-dxdd krb5kdc[26371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332763, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:39:23.488449 26477 init.cc:377] Logged in from keytab as kudu/127.23.100.1@KRBTEST.COM (short username kudu)
I20260925 10:39:23.489814 26477 webserver.cc:460] Webserver started at http://127.23.100.1:38457/ using document root <none> and password file <none>
I20260925 10:39:23.490520 26477 fs_manager.cc:359] Metadata directory not provided
I20260925 10:39:23.490654 26477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:39:23.490931 26477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:39:23.493588 26477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-0/data/instance:
uuid: "812e4f5d618d486688be540855417ff9"
format_stamp: "Formatted at 2026-09-25 10:39:23 on dist-test-slave-dxdd"
server_key: "1b1d8105824c01d21613b2887dbfb756"
server_key_iv: "b63a038ada5ff33fb81d53a87ba94cc9"
server_key_version: "encryptionkey@0"
I20260925 10:39:23.494257 26477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-0/wal/instance:
uuid: "812e4f5d618d486688be540855417ff9"
format_stamp: "Formatted at 2026-09-25 10:39:23 on dist-test-slave-dxdd"
server_key: "1b1d8105824c01d21613b2887dbfb756"
server_key_iv: "b63a038ada5ff33fb81d53a87ba94cc9"
server_key_version: "encryptionkey@0"
I20260925 10:39:23.499010 26477 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 10:39:23.502113 26492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:23.503417 26477 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 10:39:23.503587 26477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-0/wal
uuid: "812e4f5d618d486688be540855417ff9"
format_stamp: "Formatted at 2026-09-25 10:39:23 on dist-test-slave-dxdd"
server_key: "1b1d8105824c01d21613b2887dbfb756"
server_key_iv: "b63a038ada5ff33fb81d53a87ba94cc9"
server_key_version: "encryptionkey@0"
I20260925 10:39:23.503763 26477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:39:23.550591 26477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:39:23.554476 26477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:39:23.554750 26477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:39:23.555641 26477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:39:23.556862 26477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:39:23.556962 26477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:23.557116 26477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:39:23.557184 26477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:23.589215 26477 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.1:34951
I20260925 10:39:23.589263 26604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.1:34951 every 8 connection(s)
I20260925 10:39:23.590540 26477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
I20260925 10:39:23.592739 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 26477
I20260925 10:39:23.592985 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-0/wal/instance
I20260925 10:39:23.593722 23952 external_mini_cluster.cc:1428] Setting key 3137ab2fa8662bf83c3998a257959d7c
Sep 25 10:39:23 dist-test-slave-dxdd krb5kdc[26371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332763, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.1@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:39:23.613184 26607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:23.593028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:36475 (local address 127.23.100.62:34367)
0925 10:39:23.593503 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:23.593510 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:23.594644 (+ 1134us) server_negotiation.cc:407] Connection header received
0925 10:39:23.596747 (+ 2103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:23.596751 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:23.596818 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:23.596897 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:23.599026 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:23.599724 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:23.601991 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:23.602732 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:23.602882 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:23.606361 (+ 3479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:23.606370 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:23.606378 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:23.606424 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:23.609944 (+ 3520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:23.610709 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:23.610717 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:23.610722 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:23.610876 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:23.611299 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:23.611307 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:23.611312 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:23.611671 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:23.611832 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:23.612515 (+ 683us) server_negotiation.cc:299] Negotiation successful
0925 10:39:23.612742 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":237,"threads_started":1}
WARNING: no policy specified for kudu/127.23.100.2@KRBTEST.COM; defaulting to no policy
I20260925 10:39:23.613770 26608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:23.593656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:34367 (local address 127.23.100.1:36475)
0925 10:39:23.594467 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:23.594505 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:23.596456 (+ 1951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:23.597191 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:23.597206 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:23.598035 (+ 829us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:23.598862 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:23.598891 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:23.599873 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:23.599881 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:23.601861 (+ 1980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:23.601873 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:23.602931 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:23.602936 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:23.603126 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:23.603798 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:23.603823 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:23.606122 (+ 2299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:23.610100 (+ 3978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:23.610107 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:23.610118 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:23.610594 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:23.611001 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:23.611004 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:23.611006 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:23.611180 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:23.611793 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:23.611799 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:23.612127 (+ 328us) client_negotiation.cc:769] Sending connection context
0925 10:39:23.612342 (+ 215us) client_negotiation.cc:240] Negotiation successful
0925 10:39:23.612704 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":313,"threads_started":1}
I20260925 10:39:23.615442 26605 heartbeater.cc:347] Connected to a master server at 127.23.100.62:34367
I20260925 10:39:23.615762 26605 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.2@KRBTEST.COM" created.
I20260925 10:39:23.616536 26605 heartbeater.cc:511] Master 127.23.100.62:34367 requested a full tablet report, sending...
I20260925 10:39:23.618618 26416 ts_manager.cc:194] Registered new tserver with Master: 812e4f5d618d486688be540855417ff9 (127.23.100.1:34951)
I20260925 10:39:23.620039 26416 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.1@KRBTEST.COM'} at 127.23.100.1:36475
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab.
Entry for principal kudu/127.23.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab.
I20260925 10:39:23.638906 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.2:0
--local_ip_for_outbound_sockets=127.23.100.2
--webserver_interface=127.23.100.2
--webserver_port=0
--tserver_master_addrs=127.23.100.62:34367
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab
--principal=kudu/127.23.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:44265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:39:23.945513 26611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:39:23.945760 26611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:39:23.945910 26611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:39:23.953013 26611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:39:23.953107 26611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:39:23.953334 26611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.2
I20260925 10:39:23.962647 26611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:44265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab
--principal=kudu/127.23.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.2
--webserver_port=0
--tserver_master_addrs=127.23.100.62:34367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.2
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:39:23.963850 26611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:39:23.965124 26611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:39:23.978134 26616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:23.978220 26619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:23.978193 26617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:23.979285 26611 server_base.cc:1034] running on GCE node
I20260925 10:39:23.979849 26611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:39:23.980682 26611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:39:23.981896 26611 hybrid_clock.cc:648] HybridClock initialized: now 1790332763981863 us; error 48 us; skew 500 ppm
I20260925 10:39:23.982196 26611 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:39:23 dist-test-slave-dxdd krb5kdc[26371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332763, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:39:23.985556 26611 init.cc:377] Logged in from keytab as kudu/127.23.100.2@KRBTEST.COM (short username kudu)
I20260925 10:39:23.987581 26611 webserver.cc:460] Webserver started at http://127.23.100.2:43543/ using document root <none> and password file <none>
I20260925 10:39:23.988322 26611 fs_manager.cc:359] Metadata directory not provided
I20260925 10:39:23.988452 26611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:39:23.988758 26611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:39:23.991561 26611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-1/data/instance:
uuid: "01c2cf76aa0f4ac4b1741285047d97a1"
format_stamp: "Formatted at 2026-09-25 10:39:23 on dist-test-slave-dxdd"
server_key: "783b4f3c16af48147a877ef642b379d0"
server_key_iv: "68ff4fe62895c8b1ff13587df87123b9"
server_key_version: "encryptionkey@0"
I20260925 10:39:23.992311 26611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-1/wal/instance:
uuid: "01c2cf76aa0f4ac4b1741285047d97a1"
format_stamp: "Formatted at 2026-09-25 10:39:23 on dist-test-slave-dxdd"
server_key: "783b4f3c16af48147a877ef642b379d0"
server_key_iv: "68ff4fe62895c8b1ff13587df87123b9"
server_key_version: "encryptionkey@0"
I20260925 10:39:23.997779 26611 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 10:39:24.001116 26626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:24.002462 26611 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 10:39:24.002617 26611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-1/wal
uuid: "01c2cf76aa0f4ac4b1741285047d97a1"
format_stamp: "Formatted at 2026-09-25 10:39:23 on dist-test-slave-dxdd"
server_key: "783b4f3c16af48147a877ef642b379d0"
server_key_iv: "68ff4fe62895c8b1ff13587df87123b9"
server_key_version: "encryptionkey@0"
I20260925 10:39:24.002815 26611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:39:24.038978 26611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:39:24.043593 26611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:39:24.043900 26611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:39:24.044890 26611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:39:24.046294 26611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:39:24.046429 26611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:24.046586 26611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:39:24.046636 26611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:24.079732 26611 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.2:37783
I20260925 10:39:24.079802 26738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.2:37783 every 8 connection(s)
I20260925 10:39:24.080971 26611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
I20260925 10:39:24.085088 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 26611
I20260925 10:39:24.085285 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-1/wal/instance
I20260925 10:39:24.085841 23952 external_mini_cluster.cc:1428] Setting key 521165163c85623e50ad54dc689953fa
Sep 25 10:39:24 dist-test-slave-dxdd krb5kdc[26371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332763, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.2@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:39:24.103251 26607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:24.083482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:55305 (local address 127.23.100.62:34367)
0925 10:39:24.083679 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:24.083683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:24.084823 (+ 1140us) server_negotiation.cc:407] Connection header received
0925 10:39:24.087062 (+ 2239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:24.087067 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:24.087149 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:24.087251 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:24.088970 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:24.089654 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:24.092092 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:24.092529 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:24.092626 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:24.096489 (+ 3863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:24.096497 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:24.096505 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:24.096547 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:24.100255 (+ 3708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:24.100934 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:24.100943 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:24.100950 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:24.101124 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:24.101519 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:24.101527 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:24.101533 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:24.101909 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:24.102074 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:24.102844 (+ 770us) server_negotiation.cc:299] Negotiation successful
0925 10:39:24.103008 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 10:39:24.104774 26741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:24.083963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:34367 (local address 127.23.100.2:55305)
0925 10:39:24.084615 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:24.084657 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:24.086828 (+ 2171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:24.087430 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:24.087437 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:24.088000 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:24.088789 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:24.088815 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:24.089812 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:24.089820 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:24.091922 (+ 2102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:24.091938 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:24.092748 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:24.092757 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:24.093069 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:24.093859 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:24.093884 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:24.096272 (+ 2388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:24.100393 (+ 4121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:24.100399 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:24.100409 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:24.100800 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:24.101229 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:24.101232 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:24.101234 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:24.101394 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:24.102044 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:24.102049 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:24.102397 (+ 348us) client_negotiation.cc:769] Sending connection context
0925 10:39:24.102737 (+ 340us) client_negotiation.cc:240] Negotiation successful
0925 10:39:24.103308 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":254,"threads_started":1}
I20260925 10:39:24.106213 26739 heartbeater.cc:347] Connected to a master server at 127.23.100.62:34367
WARNING: no policy specified for kudu/127.23.100.3@KRBTEST.COM; defaulting to no policy
I20260925 10:39:24.106573 26739 heartbeater.cc:464] Registering TS with master...
I20260925 10:39:24.107436 26739 heartbeater.cc:511] Master 127.23.100.62:34367 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.3@KRBTEST.COM" created.
I20260925 10:39:24.108929 26416 ts_manager.cc:194] Registered new tserver with Master: 01c2cf76aa0f4ac4b1741285047d97a1 (127.23.100.2:37783)
I20260925 10:39:24.109695 26416 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.2@KRBTEST.COM'} at 127.23.100.2:55305
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab.
Entry for principal kudu/127.23.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab.
I20260925 10:39:24.130868 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.3:0
--local_ip_for_outbound_sockets=127.23.100.3
--webserver_interface=127.23.100.3
--webserver_port=0
--tserver_master_addrs=127.23.100.62:34367
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab
--principal=kudu/127.23.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:44265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:39:24.454746 26744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:39:24.455042 26744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:39:24.455216 26744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:39:24.462106 26744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:39:24.462245 26744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:39:24.462445 26744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.3
I20260925 10:39:24.471868 26744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:44265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab
--principal=kudu/127.23.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.3
--webserver_port=0
--tserver_master_addrs=127.23.100.62:34367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.3
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:39:24.473048 26744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:39:24.474330 26744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:39:24.486833 26749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:24.486847 26752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:24.486847 26750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:24.487551 26744 server_base.cc:1034] running on GCE node
I20260925 10:39:24.488152 26744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:39:24.489017 26744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:39:24.490258 26744 hybrid_clock.cc:648] HybridClock initialized: now 1790332764490209 us; error 44 us; skew 500 ppm
I20260925 10:39:24.490617 26744 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:39:24 dist-test-slave-dxdd krb5kdc[26371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332764, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:39:24.493626 26744 init.cc:377] Logged in from keytab as kudu/127.23.100.3@KRBTEST.COM (short username kudu)
I20260925 10:39:24.495069 26744 webserver.cc:460] Webserver started at http://127.23.100.3:45779/ using document root <none> and password file <none>
I20260925 10:39:24.495669 26744 fs_manager.cc:359] Metadata directory not provided
I20260925 10:39:24.495765 26744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:39:24.496013 26744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:39:24.498927 26744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-2/data/instance:
uuid: "499765d600d04994b25fb29fca23528b"
format_stamp: "Formatted at 2026-09-25 10:39:24 on dist-test-slave-dxdd"
server_key: "156809fbe82dd3467794d5b3c39c1f2f"
server_key_iv: "90180e627319d8bf6f4fe0b838e24257"
server_key_version: "encryptionkey@0"
I20260925 10:39:24.499653 26744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-2/wal/instance:
uuid: "499765d600d04994b25fb29fca23528b"
format_stamp: "Formatted at 2026-09-25 10:39:24 on dist-test-slave-dxdd"
server_key: "156809fbe82dd3467794d5b3c39c1f2f"
server_key_iv: "90180e627319d8bf6f4fe0b838e24257"
server_key_version: "encryptionkey@0"
I20260925 10:39:24.504361 26744 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 10:39:24.507982 26759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:24.509218 26744 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 10:39:24.509343 26744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-2/wal
uuid: "499765d600d04994b25fb29fca23528b"
format_stamp: "Formatted at 2026-09-25 10:39:24 on dist-test-slave-dxdd"
server_key: "156809fbe82dd3467794d5b3c39c1f2f"
server_key_iv: "90180e627319d8bf6f4fe0b838e24257"
server_key_version: "encryptionkey@0"
I20260925 10:39:24.509493 26744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:39:24.530202 26744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:39:24.534202 26744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:39:24.534485 26744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:39:24.535341 26744 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:39:24.536530 26744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:39:24.536608 26744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:24.536718 26744 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:39:24.536783 26744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:24.568794 26744 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.3:33247
I20260925 10:39:24.568878 26871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.3:33247 every 8 connection(s)
I20260925 10:39:24.570037 26744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
I20260925 10:39:24.577693 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 26744
I20260925 10:39:24.577881 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-2/wal/instance
I20260925 10:39:24.578347 23952 external_mini_cluster.cc:1428] Setting key 3f4223d1c207f96c5dbeff99e9b63505
Sep 25 10:39:24 dist-test-slave-dxdd krb5kdc[26371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332764, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.3@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:39:24.590167 26607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:24.572485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:41313 (local address 127.23.100.62:34367)
0925 10:39:24.572662 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:24.572666 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:24.573830 (+ 1164us) server_negotiation.cc:407] Connection header received
0925 10:39:24.575697 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:24.575702 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:24.575821 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:24.575933 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:24.577422 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:24.578299 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:24.580171 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:24.580783 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:24.580905 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:24.584415 (+ 3510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:24.584420 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:24.584423 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:24.584455 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:24.587102 (+ 2647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:24.587732 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:24.587736 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:24.587738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:24.587857 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:24.588203 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:24.588206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:24.588208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:24.588509 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:24.588685 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:24.589545 (+ 860us) server_negotiation.cc:299] Negotiation successful
0925 10:39:24.589778 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 10:39:24.590848 26874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:24.572986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:34367 (local address 127.23.100.3:41313)
0925 10:39:24.573653 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:24.573701 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:24.575418 (+ 1717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:24.576188 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:24.576196 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:24.576694 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:24.577229 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:24.577245 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:24.578517 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:24.578521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:24.580002 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:24.580013 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:24.581029 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:24.581039 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:24.581276 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:24.581962 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:24.581988 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:24.584215 (+ 2227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:24.587222 (+ 3007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:24.587228 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:24.587239 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:24.587636 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:24.587946 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:24.587949 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:24.587951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:24.588114 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:24.588654 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:24.588662 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:24.589016 (+ 354us) client_negotiation.cc:769] Sending connection context
0925 10:39:24.589280 (+ 264us) client_negotiation.cc:240] Negotiation successful
0925 10:39:24.589693 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":252,"threads_started":1}
I20260925 10:39:24.592346 26872 heartbeater.cc:347] Connected to a master server at 127.23.100.62:34367
I20260925 10:39:24.592758 26872 heartbeater.cc:464] Registering TS with master...
I20260925 10:39:24.593638 26872 heartbeater.cc:511] Master 127.23.100.62:34367 requested a full tablet report, sending...
I20260925 10:39:24.594996 26416 ts_manager.cc:194] Registered new tserver with Master: 499765d600d04994b25fb29fca23528b (127.23.100.3:33247)
I20260925 10:39:24.595738 26416 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.3@KRBTEST.COM'} at 127.23.100.3:41313
I20260925 10:39:24.605736 23952 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:39:24.623560 26605 heartbeater.cc:503] Master 127.23.100.62:34367 was elected leader, sending a full tablet report...
W20260925 10:39:24.636101 26607 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:39:24.633538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49004 (local address 127.23.100.62:34367)
0925 10:39:24.633742 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:24.633751 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:24.634210 (+ 459us) server_negotiation.cc:407] Connection header received
0925 10:39:24.634447 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:24.634452 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:24.634525 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:24.634633 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:24.635808 (+ 1175us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49004: BlockingRecv error: recv got EOF from 127.0.0.1:49004 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 10:39:24.639446 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 26477
W20260925 10:39:24.642095 23952 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 26477: Not authorized: Client connection negotiation failed: client connection to 127.23.100.1:34951: server requires authentication, but client does not have Kerberos credentials available
I20260925 10:39:24.642282 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 26477
I20260925 10:39:24.658363 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 26611
W20260925 10:39:24.661005 23952 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 26611: Not authorized: Client connection negotiation failed: client connection to 127.23.100.2:37783: server requires authentication, but client does not have Kerberos credentials available
I20260925 10:39:24.661175 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 26611
I20260925 10:39:24.677145 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 26744
W20260925 10:39:24.680047 23952 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 26744: Not authorized: Client connection negotiation failed: client connection to 127.23.100.3:33247: server requires authentication, but client does not have Kerberos credentials available
I20260925 10:39:24.680168 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 26744
I20260925 10:39:24.694811 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 26384
I20260925 10:39:24.695904 26446 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:39:24.798950 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 26384
2026-09-25T10:39:24Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3482 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790332740062128-23952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:39:24 dist-test-slave-dxdd krb5kdc[26889](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:39:24 dist-test-slave-dxdd krb5kdc[26889](info): set up 2 sockets
Sep 25 10:39:24 dist-test-slave-dxdd krb5kdc[26889](info): commencing operation
krb5kdc: starting...
W20260925 10:39:25.882664 23952 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.007s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Sep 25 10:39:26 dist-test-slave-dxdd krb5kdc[26889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:39:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:39:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
I20260925 10:39:26.069417 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:41073
--webserver_interface=127.23.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:34941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.62:41073
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:39:26.379796 26902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:39:26.380077 26902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:39:26.380167 26902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:39:26.387084 26902 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:39:26.387176 26902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:39:26.387218 26902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:39:26.387274 26902 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:39:26.387317 26902 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:39:26.396618 26902 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:34941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790332740062128-23952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.62:41073
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:41073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:39:26.397754 26902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:39:26.398936 26902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:39:26.410148 26907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:26.410087 26910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:26.410120 26908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:26.411116 26902 server_base.cc:1034] running on GCE node
I20260925 10:39:26.411892 26902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:39:26.413143 26902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:39:26.414393 26902 hybrid_clock.cc:648] HybridClock initialized: now 1790332766414362 us; error 44 us; skew 500 ppm
I20260925 10:39:26.414716 26902 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:39:26 dist-test-slave-dxdd krb5kdc[26889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332766, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:39:26.422219 26902 init.cc:377] Logged in from keytab as kudu/127.23.100.62@KRBTEST.COM (short username kudu)
I20260925 10:39:26.423641 26902 webserver.cc:460] Webserver started at http://127.23.100.62:40835/ using document root <none> and password file <none>
I20260925 10:39:26.424244 26902 fs_manager.cc:359] Metadata directory not provided
I20260925 10:39:26.424335 26902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:39:26.424645 26902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:39:26.427380 26902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790332740062128-23952-0/minicluster-data/master-0/data/instance:
uuid: "eab9f01dbf774b9e8825b77fdd6e1e98"
format_stamp: "Formatted at 2026-09-25 10:39:26 on dist-test-slave-dxdd"
server_key: "49aebcc0bb02710d54e2dce773c309bd"
server_key_iv: "00655338341ea9101a74aad618339e19"
server_key_version: "encryptionkey@0"
I20260925 10:39:26.428064 26902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790332740062128-23952-0/minicluster-data/master-0/wal/instance:
uuid: "eab9f01dbf774b9e8825b77fdd6e1e98"
format_stamp: "Formatted at 2026-09-25 10:39:26 on dist-test-slave-dxdd"
server_key: "49aebcc0bb02710d54e2dce773c309bd"
server_key_iv: "00655338341ea9101a74aad618339e19"
server_key_version: "encryptionkey@0"
I20260925 10:39:26.432729 26902 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 10:39:26.435817 26917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:26.437021 26902 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 10:39:26.437189 26902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790332740062128-23952-0/minicluster-data/master-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790332740062128-23952-0/minicluster-data/master-0/wal
uuid: "eab9f01dbf774b9e8825b77fdd6e1e98"
format_stamp: "Formatted at 2026-09-25 10:39:26 on dist-test-slave-dxdd"
server_key: "49aebcc0bb02710d54e2dce773c309bd"
server_key_iv: "00655338341ea9101a74aad618339e19"
server_key_version: "encryptionkey@0"
I20260925 10:39:26.437381 26902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790332740062128-23952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790332740062128-23952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790332740062128-23952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:39:26.451164 26902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:39:26.454842 26902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:39:26.455138 26902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:39:26.473048 26902 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.62:41073
I20260925 10:39:26.473091 26968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.62:41073 every 8 connection(s)
I20260925 10:39:26.474362 26902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
I20260925 10:39:26.475715 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 26902
I20260925 10:39:26.475939 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790332740062128-23952-0/minicluster-data/master-0/wal/instance
I20260925 10:39:26.476728 23952 external_mini_cluster.cc:1428] Setting key 638496ea91285b277ec8f6cd59e92397
I20260925 10:39:26.478981 26969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 10:39:26 dist-test-slave-dxdd krb5kdc[26889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:39:26.491813 26969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eab9f01dbf774b9e8825b77fdd6e1e98: Bootstrap starting.
I20260925 10:39:26.495906 26969 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eab9f01dbf774b9e8825b77fdd6e1e98: Neither blocks nor log segments found. Creating new log.
I20260925 10:39:26.497146 26969 log.cc:824] T 00000000000000000000000000000000 P eab9f01dbf774b9e8825b77fdd6e1e98: Log is configured to *not* fsync() on all Append() calls
I20260925 10:39:26.500038 26972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:26.478589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58830 (local address 127.23.100.62:41073)
0925 10:39:26.479404 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:26.479415 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:26.479452 (+ 37us) server_negotiation.cc:407] Connection header received
0925 10:39:26.481017 (+ 1565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:26.481037 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:26.481378 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:26.481687 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:26.483555 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:26.484796 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:26.486540 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:26.487167 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:26.487283 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:26.491748 (+ 4465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:26.491758 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:26.491776 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:26.491831 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:26.495651 (+ 3820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:26.496335 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:26.496348 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:26.496355 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:26.496542 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:26.496889 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:26.496897 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:26.496903 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:26.497593 (+ 690us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:26.497923 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:26.498381 (+ 458us) server_negotiation.cc:299] Negotiation successful
0925 10:39:26.498806 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":436,"threads_started":1}
I20260925 10:39:26.500381 26969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eab9f01dbf774b9e8825b77fdd6e1e98: No bootstrap required, opened a new log
I20260925 10:39:26.503773 23952 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:39:26.508224 26969 raft_consensus.cc:348] T 00000000000000000000000000000000 P eab9f01dbf774b9e8825b77fdd6e1e98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eab9f01dbf774b9e8825b77fdd6e1e98" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 41073 } }
I20260925 10:39:26.508582 26969 raft_consensus.cc:374] T 00000000000000000000000000000000 P eab9f01dbf774b9e8825b77fdd6e1e98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:39:26.508688 26969 raft_consensus.cc:729] T 00000000000000000000000000000000 P eab9f01dbf774b9e8825b77fdd6e1e98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eab9f01dbf774b9e8825b77fdd6e1e98, State: Initialized, Role: FOLLOWER
I20260925 10:39:26.509272 26969 consensus_queue.cc:260] T 00000000000000000000000000000000 P eab9f01dbf774b9e8825b77fdd6e1e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eab9f01dbf774b9e8825b77fdd6e1e98" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 41073 } }
I20260925 10:39:26.509508 26969 raft_consensus.cc:388] T 00000000000000000000000000000000 P eab9f01dbf774b9e8825b77fdd6e1e98 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:39:26.509665 26969 raft_consensus.cc:482] T 00000000000000000000000000000000 P eab9f01dbf774b9e8825b77fdd6e1e98 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:39:26.509853 26969 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eab9f01dbf774b9e8825b77fdd6e1e98 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:39:26.512038 26969 raft_consensus.cc:504] T 00000000000000000000000000000000 P eab9f01dbf774b9e8825b77fdd6e1e98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eab9f01dbf774b9e8825b77fdd6e1e98" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 41073 } }
I20260925 10:39:26.512516 26969 leader_election.cc:302] T 00000000000000000000000000000000 P eab9f01dbf774b9e8825b77fdd6e1e98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eab9f01dbf774b9e8825b77fdd6e1e98; no voters:
I20260925 10:39:26.513137 26969 leader_election.cc:288] T 00000000000000000000000000000000 P eab9f01dbf774b9e8825b77fdd6e1e98 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:39:26.513227 26975 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eab9f01dbf774b9e8825b77fdd6e1e98 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:39:26.514413 26975 raft_consensus.cc:686] T 00000000000000000000000000000000 P eab9f01dbf774b9e8825b77fdd6e1e98 [term 1 LEADER]: Becoming Leader. State: Replica: eab9f01dbf774b9e8825b77fdd6e1e98, State: Running, Role: LEADER
I20260925 10:39:26.515045 26975 consensus_queue.cc:237] T 00000000000000000000000000000000 P eab9f01dbf774b9e8825b77fdd6e1e98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eab9f01dbf774b9e8825b77fdd6e1e98" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 41073 } }
I20260925 10:39:26.515496 26969 sys_catalog.cc:564] T 00000000000000000000000000000000 P eab9f01dbf774b9e8825b77fdd6e1e98 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:39:26.517951 26976 sys_catalog.cc:455] T 00000000000000000000000000000000 P eab9f01dbf774b9e8825b77fdd6e1e98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eab9f01dbf774b9e8825b77fdd6e1e98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eab9f01dbf774b9e8825b77fdd6e1e98" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 41073 } } }
I20260925 10:39:26.518045 26977 sys_catalog.cc:455] T 00000000000000000000000000000000 P eab9f01dbf774b9e8825b77fdd6e1e98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eab9f01dbf774b9e8825b77fdd6e1e98. Latest consensus state: current_term: 1 leader_uuid: "eab9f01dbf774b9e8825b77fdd6e1e98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eab9f01dbf774b9e8825b77fdd6e1e98" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 41073 } } }
I20260925 10:39:26.518383 26976 sys_catalog.cc:458] T 00000000000000000000000000000000 P eab9f01dbf774b9e8825b77fdd6e1e98 [sys.catalog]: This master's current role is: LEADER
I20260925 10:39:26.518429 26977 sys_catalog.cc:458] T 00000000000000000000000000000000 P eab9f01dbf774b9e8825b77fdd6e1e98 [sys.catalog]: This master's current role is: LEADER
I20260925 10:39:26.519327 26981 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:39:26.537456 26981 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:39:26.549932 26981 catalog_manager.cc:1347] Generated new cluster ID: 5ca40035f56a48c4b7212f6e9770863c
I20260925 10:39:26.550081 26981 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:39:26.564391 26981 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:39:26.566063 26981 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:39:26.572279 26972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:26.564205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58832 (local address 127.23.100.62:41073)
0925 10:39:26.564410 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:26.564420 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:26.565192 (+ 772us) server_negotiation.cc:407] Connection header received
0925 10:39:26.565435 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:26.565440 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:26.566069 (+ 629us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:26.566189 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:26.567208 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:26.567948 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:26.569202 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:26.569599 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:26.569745 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:26.570822 (+ 1077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:26.570828 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:26.570831 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:39:26.571451 (+ 620us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:26.571491 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:39:26.571520 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:26.571651 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:26.571782 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 10:39:26.571870 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 10:39:26.574561 26981 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eab9f01dbf774b9e8825b77fdd6e1e98: Generated new TSK 0
I20260925 10:39:26.575359 26981 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:39:26.582971 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 26902
I20260925 10:39:26.584002 26964 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:39:26.669508 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 26902
2026-09-25T10:39:26Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1863 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:39:26 dist-test-slave-dxdd krb5kdc[27003](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:39:26 dist-test-slave-dxdd krb5kdc[27003](info): set up 2 sockets
Sep 25 10:39:26 dist-test-slave-dxdd krb5kdc[27003](info): commencing operation
krb5kdc: starting...
W20260925 10:39:27.722676 23952 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.005s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Sep 25 10:39:27 dist-test-slave-dxdd krb5kdc[27003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:39:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:39:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
I20260925 10:39:27.905691 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:38981
--webserver_interface=127.23.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:42299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.62:38981
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:39:28.210516 27016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:39:28.210959 27016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:39:28.211163 27016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:39:28.219409 27016 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:39:28.219550 27016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:39:28.219668 27016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:39:28.219761 27016 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:39:28.219844 27016 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:39:28.229490 27016 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:42299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.62:38981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:38981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:39:28.230774 27016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:39:28.232010 27016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:39:28.243407 27021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:28.243402 27024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:28.243716 27022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:28.244058 27016 server_base.cc:1034] running on GCE node
I20260925 10:39:28.244743 27016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:39:28.245935 27016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:39:28.247181 27016 hybrid_clock.cc:648] HybridClock initialized: now 1790332768247134 us; error 44 us; skew 500 ppm
I20260925 10:39:28.247455 27016 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:39:28 dist-test-slave-dxdd krb5kdc[27003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332768, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:39:28.250423 27016 init.cc:377] Logged in from keytab as kudu/127.23.100.62@KRBTEST.COM (short username kudu)
I20260925 10:39:28.251897 27016 webserver.cc:460] Webserver started at http://127.23.100.62:38187/ using document root <none> and password file <none>
I20260925 10:39:28.252529 27016 fs_manager.cc:359] Metadata directory not provided
I20260925 10:39:28.252672 27016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:39:28.252962 27016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:39:28.255805 27016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/master-0/data/instance:
uuid: "cd47992f9ad04e6f8fc9a344f327257f"
format_stamp: "Formatted at 2026-09-25 10:39:28 on dist-test-slave-dxdd"
server_key: "d5c1275133739cc734d076e1b73a9030"
server_key_iv: "b38f6f7a772c5c239146c0bd30e49112"
server_key_version: "encryptionkey@0"
I20260925 10:39:28.256527 27016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/master-0/wal/instance:
uuid: "cd47992f9ad04e6f8fc9a344f327257f"
format_stamp: "Formatted at 2026-09-25 10:39:28 on dist-test-slave-dxdd"
server_key: "d5c1275133739cc734d076e1b73a9030"
server_key_iv: "b38f6f7a772c5c239146c0bd30e49112"
server_key_version: "encryptionkey@0"
I20260925 10:39:28.261410 27016 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 10:39:28.264649 27031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:28.265856 27016 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 10:39:28.266026 27016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/master-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/master-0/wal
uuid: "cd47992f9ad04e6f8fc9a344f327257f"
format_stamp: "Formatted at 2026-09-25 10:39:28 on dist-test-slave-dxdd"
server_key: "d5c1275133739cc734d076e1b73a9030"
server_key_iv: "b38f6f7a772c5c239146c0bd30e49112"
server_key_version: "encryptionkey@0"
I20260925 10:39:28.266202 27016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:39:28.281214 27016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:39:28.284864 27016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:39:28.285142 27016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:39:28.302930 27016 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.62:38981
I20260925 10:39:28.302966 27082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.62:38981 every 8 connection(s)
I20260925 10:39:28.304262 27016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
I20260925 10:39:28.307734 27083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:39:28.312664 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 27016
I20260925 10:39:28.312879 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/master-0/wal/instance
I20260925 10:39:28.313484 23952 external_mini_cluster.cc:1428] Setting key ffeb0d7b1959b6ed1efa5ccb9d10ba1a
I20260925 10:39:28.319581 27083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd47992f9ad04e6f8fc9a344f327257f: Bootstrap starting.
I20260925 10:39:28.323346 27083 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd47992f9ad04e6f8fc9a344f327257f: Neither blocks nor log segments found. Creating new log.
I20260925 10:39:28.324342 27083 log.cc:824] T 00000000000000000000000000000000 P cd47992f9ad04e6f8fc9a344f327257f: Log is configured to *not* fsync() on all Append() calls
I20260925 10:39:28.327018 27083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd47992f9ad04e6f8fc9a344f327257f: No bootstrap required, opened a new log
Sep 25 10:39:28 dist-test-slave-dxdd krb5kdc[27003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:39:28.334702 27083 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd47992f9ad04e6f8fc9a344f327257f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd47992f9ad04e6f8fc9a344f327257f" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 38981 } }
I20260925 10:39:28.335052 27083 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd47992f9ad04e6f8fc9a344f327257f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:39:28.335191 27083 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd47992f9ad04e6f8fc9a344f327257f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd47992f9ad04e6f8fc9a344f327257f, State: Initialized, Role: FOLLOWER
I20260925 10:39:28.335917 27083 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd47992f9ad04e6f8fc9a344f327257f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd47992f9ad04e6f8fc9a344f327257f" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 38981 } }
I20260925 10:39:28.336171 27083 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd47992f9ad04e6f8fc9a344f327257f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:39:28.336323 27083 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd47992f9ad04e6f8fc9a344f327257f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:39:28.336508 27083 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd47992f9ad04e6f8fc9a344f327257f [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:39:28.336699 27086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:28.315486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33698 (local address 127.23.100.62:38981)
0925 10:39:28.316451 (+ 965us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:28.316465 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:28.316528 (+ 63us) server_negotiation.cc:407] Connection header received
0925 10:39:28.319010 (+ 2482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:28.319037 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:28.319507 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:28.319935 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:28.321125 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:28.323099 (+ 1974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:28.324311 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:28.325275 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:28.325472 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:28.329413 (+ 3941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:28.329420 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:28.329433 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:28.329474 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:28.332606 (+ 3132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:28.333303 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:28.333312 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:28.333318 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:28.333500 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:28.333883 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:28.333888 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:28.333901 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:28.334458 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:28.334775 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:28.335210 (+ 435us) server_negotiation.cc:299] Negotiation successful
0925 10:39:28.335603 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":709,"thread_start_us":423,"threads_started":1}
I20260925 10:39:28.339458 27083 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd47992f9ad04e6f8fc9a344f327257f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd47992f9ad04e6f8fc9a344f327257f" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 38981 } }
I20260925 10:39:28.339953 27083 leader_election.cc:302] T 00000000000000000000000000000000 P cd47992f9ad04e6f8fc9a344f327257f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd47992f9ad04e6f8fc9a344f327257f; no voters:
I20260925 10:39:28.340472 27083 leader_election.cc:288] T 00000000000000000000000000000000 P cd47992f9ad04e6f8fc9a344f327257f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:39:28.340606 27088 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd47992f9ad04e6f8fc9a344f327257f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:39:28.341305 27088 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd47992f9ad04e6f8fc9a344f327257f [term 1 LEADER]: Becoming Leader. State: Replica: cd47992f9ad04e6f8fc9a344f327257f, State: Running, Role: LEADER
I20260925 10:39:28.341769 27088 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd47992f9ad04e6f8fc9a344f327257f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd47992f9ad04e6f8fc9a344f327257f" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 38981 } }
I20260925 10:39:28.342618 27083 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd47992f9ad04e6f8fc9a344f327257f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:39:28.344833 27090 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd47992f9ad04e6f8fc9a344f327257f [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd47992f9ad04e6f8fc9a344f327257f. Latest consensus state: current_term: 1 leader_uuid: "cd47992f9ad04e6f8fc9a344f327257f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd47992f9ad04e6f8fc9a344f327257f" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 38981 } } }
I20260925 10:39:28.345248 27089 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd47992f9ad04e6f8fc9a344f327257f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd47992f9ad04e6f8fc9a344f327257f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd47992f9ad04e6f8fc9a344f327257f" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 38981 } } }
I20260925 10:39:28.345573 27089 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd47992f9ad04e6f8fc9a344f327257f [sys.catalog]: This master's current role is: LEADER
I20260925 10:39:28.345245 27090 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd47992f9ad04e6f8fc9a344f327257f [sys.catalog]: This master's current role is: LEADER
I20260925 10:39:28.347433 27097 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:39:28.352639 27097 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:39:28.361142 27097 catalog_manager.cc:1347] Generated new cluster ID: c1f5f7fbbf244dc593b5ade51120463b
I20260925 10:39:28.361258 27097 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:39:28.379016 27097 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:39:28.380296 27097 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:39:28.392285 27097 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd47992f9ad04e6f8fc9a344f327257f: Generated new TSK 0
I20260925 10:39:28.393124 27097 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab.
Entry for principal kudu/127.23.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab.
I20260925 10:39:28.439033 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.1:0
--local_ip_for_outbound_sockets=127.23.100.1
--webserver_interface=127.23.100.1
--webserver_port=0
--tserver_master_addrs=127.23.100.62:38981
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab
--principal=kudu/127.23.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:42299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:39:28.751888 27109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:39:28.752147 27109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:39:28.752326 27109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:39:28.759451 27109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:39:28.759557 27109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:39:28.759761 27109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.1
I20260925 10:39:28.769382 27109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:42299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab
--principal=kudu/127.23.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.1
--webserver_port=0
--tserver_master_addrs=127.23.100.62:38981
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.1
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:39:28.770823 27109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:39:28.772112 27109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:39:28.784502 27114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:28.784561 27115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:28.784472 27117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:28.785230 27109 server_base.cc:1034] running on GCE node
I20260925 10:39:28.785732 27109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:39:28.786613 27109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:39:28.787813 27109 hybrid_clock.cc:648] HybridClock initialized: now 1790332768787744 us; error 60 us; skew 500 ppm
I20260925 10:39:28.788129 27109 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:39:28 dist-test-slave-dxdd krb5kdc[27003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332768, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:39:28.791406 27109 init.cc:377] Logged in from keytab as kudu/127.23.100.1@KRBTEST.COM (short username kudu)
I20260925 10:39:28.792837 27109 webserver.cc:460] Webserver started at http://127.23.100.1:40393/ using document root <none> and password file <none>
I20260925 10:39:28.793479 27109 fs_manager.cc:359] Metadata directory not provided
I20260925 10:39:28.793627 27109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:39:28.793937 27109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:39:28.796767 27109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-0/data/instance:
uuid: "6def0c3ba33d4d5d9908c4b0e142f362"
format_stamp: "Formatted at 2026-09-25 10:39:28 on dist-test-slave-dxdd"
server_key: "93fc2c9a96b2767f32ed2d346216c206"
server_key_iv: "afbb7825d7c1772142f8e435575999fb"
server_key_version: "encryptionkey@0"
I20260925 10:39:28.797497 27109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-0/wal/instance:
uuid: "6def0c3ba33d4d5d9908c4b0e142f362"
format_stamp: "Formatted at 2026-09-25 10:39:28 on dist-test-slave-dxdd"
server_key: "93fc2c9a96b2767f32ed2d346216c206"
server_key_iv: "afbb7825d7c1772142f8e435575999fb"
server_key_version: "encryptionkey@0"
I20260925 10:39:28.802845 27109 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260925 10:39:28.806175 27124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:28.807518 27109 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:39:28.807698 27109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-0/wal
uuid: "6def0c3ba33d4d5d9908c4b0e142f362"
format_stamp: "Formatted at 2026-09-25 10:39:28 on dist-test-slave-dxdd"
server_key: "93fc2c9a96b2767f32ed2d346216c206"
server_key_iv: "afbb7825d7c1772142f8e435575999fb"
server_key_version: "encryptionkey@0"
I20260925 10:39:28.807893 27109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:39:28.818228 27109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:39:28.822293 27109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:39:28.822629 27109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:39:28.823478 27109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:39:28.824724 27109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:39:28.824838 27109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:28.824988 27109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:39:28.825057 27109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:28.857861 27109 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.1:45069
I20260925 10:39:28.857896 27236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.1:45069 every 8 connection(s)
I20260925 10:39:28.859212 27109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
I20260925 10:39:28.866799 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 27109
I20260925 10:39:28.867022 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-0/wal/instance
I20260925 10:39:28.867640 23952 external_mini_cluster.cc:1428] Setting key b9d606b0bc985c5518c7071e483ce82c
Sep 25 10:39:28 dist-test-slave-dxdd krb5kdc[27003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332768, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.1@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:39:28.880832 27239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:28.861633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:36569 (local address 127.23.100.62:38981)
0925 10:39:28.862221 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:28.862225 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:28.863395 (+ 1170us) server_negotiation.cc:407] Connection header received
0925 10:39:28.865069 (+ 1674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:28.865074 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:28.865162 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:28.865278 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:28.866862 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:28.867706 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:28.869754 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:28.870186 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:28.870353 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:28.874143 (+ 3790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:28.874158 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:28.874161 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:28.874189 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:28.877659 (+ 3470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:28.878367 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:28.878375 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:28.878382 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:28.878530 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:28.878934 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:28.878941 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:28.878947 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:28.879344 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:28.879489 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:28.880121 (+ 632us) server_negotiation.cc:299] Negotiation successful
0925 10:39:28.880374 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":212,"threads_started":1}
I20260925 10:39:28.881371 27240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:28.862535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:38981 (local address 127.23.100.1:36569)
0925 10:39:28.863232 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:28.863273 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:28.864822 (+ 1549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:28.865493 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:28.865501 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:28.865962 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:28.866672 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:28.866699 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:28.867863 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:28.867868 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:28.869615 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:28.869630 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:28.870432 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:28.870441 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:28.870680 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:28.871332 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:28.871353 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:28.873866 (+ 2513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:28.877791 (+ 3925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:28.877798 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:28.877809 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:28.878209 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:28.878639 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:28.878642 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:28.878644 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:28.878823 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:28.879465 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:28.879471 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:28.879766 (+ 295us) client_negotiation.cc:769] Sending connection context
0925 10:39:28.879975 (+ 209us) client_negotiation.cc:240] Negotiation successful
0925 10:39:28.880353 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":264,"threads_started":1}
I20260925 10:39:28.882851 27237 heartbeater.cc:347] Connected to a master server at 127.23.100.62:38981
I20260925 10:39:28.883234 27237 heartbeater.cc:464] Registering TS with master...
I20260925 10:39:28.883924 27237 heartbeater.cc:511] Master 127.23.100.62:38981 requested a full tablet report, sending...
I20260925 10:39:28.885910 27048 ts_manager.cc:194] Registered new tserver with Master: 6def0c3ba33d4d5d9908c4b0e142f362 (127.23.100.1:45069)
WARNING: no policy specified for kudu/127.23.100.2@KRBTEST.COM; defaulting to no policy
I20260925 10:39:28.887442 27048 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.1@KRBTEST.COM'} at 127.23.100.1:36569
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab.
Entry for principal kudu/127.23.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab.
I20260925 10:39:28.910888 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.2:0
--local_ip_for_outbound_sockets=127.23.100.2
--webserver_interface=127.23.100.2
--webserver_port=0
--tserver_master_addrs=127.23.100.62:38981
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab
--principal=kudu/127.23.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:42299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:39:29.219486 27243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:39:29.219803 27243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:39:29.219956 27243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:39:29.227458 27243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:39:29.227596 27243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:39:29.227787 27243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.2
I20260925 10:39:29.237422 27243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:42299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab
--principal=kudu/127.23.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.2
--webserver_port=0
--tserver_master_addrs=127.23.100.62:38981
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.2
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:39:29.238770 27243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:39:29.240028 27243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:39:29.252383 27251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:29.252447 27249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:29.252451 27248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:29.258574 27243 server_base.cc:1034] running on GCE node
I20260925 10:39:29.259090 27243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:39:29.259894 27243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:39:29.261080 27243 hybrid_clock.cc:648] HybridClock initialized: now 1790332769261020 us; error 60 us; skew 500 ppm
I20260925 10:39:29.261363 27243 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:39:29 dist-test-slave-dxdd krb5kdc[27003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332769, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:39:29.264292 27243 init.cc:377] Logged in from keytab as kudu/127.23.100.2@KRBTEST.COM (short username kudu)
I20260925 10:39:29.265607 27243 webserver.cc:460] Webserver started at http://127.23.100.2:41099/ using document root <none> and password file <none>
I20260925 10:39:29.266243 27243 fs_manager.cc:359] Metadata directory not provided
I20260925 10:39:29.266409 27243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:39:29.266819 27243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:39:29.269534 27243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-1/data/instance:
uuid: "82bc306bf9f448a09e6e2aaa6b96f3d6"
format_stamp: "Formatted at 2026-09-25 10:39:29 on dist-test-slave-dxdd"
server_key: "482e6f63cb313d8184e80311d960ca97"
server_key_iv: "97c148070d1b5aa2051bd4049102a4e5"
server_key_version: "encryptionkey@0"
I20260925 10:39:29.270244 27243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-1/wal/instance:
uuid: "82bc306bf9f448a09e6e2aaa6b96f3d6"
format_stamp: "Formatted at 2026-09-25 10:39:29 on dist-test-slave-dxdd"
server_key: "482e6f63cb313d8184e80311d960ca97"
server_key_iv: "97c148070d1b5aa2051bd4049102a4e5"
server_key_version: "encryptionkey@0"
I20260925 10:39:29.274992 27243 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 10:39:29.278482 27258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:29.279680 27243 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260925 10:39:29.279838 27243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-1/wal
uuid: "82bc306bf9f448a09e6e2aaa6b96f3d6"
format_stamp: "Formatted at 2026-09-25 10:39:29 on dist-test-slave-dxdd"
server_key: "482e6f63cb313d8184e80311d960ca97"
server_key_iv: "97c148070d1b5aa2051bd4049102a4e5"
server_key_version: "encryptionkey@0"
I20260925 10:39:29.280030 27243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:39:29.309950 27243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:39:29.314013 27243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:39:29.314293 27243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:39:29.315182 27243 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:39:29.316447 27243 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:39:29.316515 27243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:29.316689 27243 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:39:29.316740 27243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:29.348892 27243 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.2:39465
I20260925 10:39:29.348938 27370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.2:39465 every 8 connection(s)
I20260925 10:39:29.350128 27243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
I20260925 10:39:29.357978 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 27243
I20260925 10:39:29.358161 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-1/wal/instance
I20260925 10:39:29.358803 23952 external_mini_cluster.cc:1428] Setting key 62044549e11b17abaec2293bf34ae0bd
Sep 25 10:39:29 dist-test-slave-dxdd krb5kdc[27003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332769, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.2@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:39:29.370669 27239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:29.352436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:32833 (local address 127.23.100.62:38981)
0925 10:39:29.352601 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:29.352606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:29.353696 (+ 1090us) server_negotiation.cc:407] Connection header received
0925 10:39:29.355428 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:29.355432 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:29.355497 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:29.355585 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:29.356928 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:29.357626 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:29.359928 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:29.360499 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:29.360643 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:29.364678 (+ 4035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:29.364683 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:29.364686 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:29.364718 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:29.367408 (+ 2690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:29.368089 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:29.368097 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:29.368104 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:29.368292 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:29.368668 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:29.368694 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:29.368700 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:29.369095 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:29.369254 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:29.369916 (+ 662us) server_negotiation.cc:299] Negotiation successful
0925 10:39:29.370193 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 10:39:29.371356 27373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:29.352912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:38981 (local address 127.23.100.2:32833)
0925 10:39:29.353542 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:29.353585 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:29.355185 (+ 1600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:29.355786 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:29.355794 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:29.356262 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:29.356784 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:29.356799 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:29.357802 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:29.357808 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:29.359770 (+ 1962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:29.359784 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:29.360716 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:29.360722 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:29.360979 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:29.361655 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:29.361688 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:29.364464 (+ 2776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:29.367546 (+ 3082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:29.367570 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:29.367580 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:29.367964 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:29.368394 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:29.368397 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:29.368399 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:29.368554 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:29.369212 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:29.369217 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:29.369491 (+ 274us) client_negotiation.cc:769] Sending connection context
0925 10:39:29.369817 (+ 326us) client_negotiation.cc:240] Negotiation successful
0925 10:39:29.370213 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":252,"threads_started":1}
I20260925 10:39:29.372756 27371 heartbeater.cc:347] Connected to a master server at 127.23.100.62:38981
I20260925 10:39:29.373050 27371 heartbeater.cc:464] Registering TS with master...
I20260925 10:39:29.373919 27371 heartbeater.cc:511] Master 127.23.100.62:38981 requested a full tablet report, sending...
I20260925 10:39:29.375435 27048 ts_manager.cc:194] Registered new tserver with Master: 82bc306bf9f448a09e6e2aaa6b96f3d6 (127.23.100.2:39465)
I20260925 10:39:29.376219 27048 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.2@KRBTEST.COM'} at 127.23.100.2:32833
WARNING: no policy specified for kudu/127.23.100.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab.
Entry for principal kudu/127.23.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab.
I20260925 10:39:29.401113 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.3:0
--local_ip_for_outbound_sockets=127.23.100.3
--webserver_interface=127.23.100.3
--webserver_port=0
--tserver_master_addrs=127.23.100.62:38981
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab
--principal=kudu/127.23.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:42299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:39:29.707109 27376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:39:29.707355 27376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:39:29.707527 27376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:39:29.714605 27376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:39:29.714694 27376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:39:29.714877 27376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.3
I20260925 10:39:29.723831 27376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:42299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab
--principal=kudu/127.23.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.3
--webserver_port=0
--tserver_master_addrs=127.23.100.62:38981
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.3
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:39:29.724901 27376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:39:29.726079 27376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:39:29.738559 27382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:29.738511 27381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:29.738596 27384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:29.739447 27376 server_base.cc:1034] running on GCE node
I20260925 10:39:29.739979 27376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:39:29.740821 27376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:39:29.742012 27376 hybrid_clock.cc:648] HybridClock initialized: now 1790332769741976 us; error 28 us; skew 500 ppm
I20260925 10:39:29.742357 27376 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:39:29 dist-test-slave-dxdd krb5kdc[27003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332769, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:39:29.745250 27376 init.cc:377] Logged in from keytab as kudu/127.23.100.3@KRBTEST.COM (short username kudu)
I20260925 10:39:29.746738 27376 webserver.cc:460] Webserver started at http://127.23.100.3:39609/ using document root <none> and password file <none>
I20260925 10:39:29.747382 27376 fs_manager.cc:359] Metadata directory not provided
I20260925 10:39:29.747542 27376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:39:29.747828 27376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:39:29.750515 27376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-2/data/instance:
uuid: "87f1df5d2054405d8b6296d60029de6e"
format_stamp: "Formatted at 2026-09-25 10:39:29 on dist-test-slave-dxdd"
server_key: "c26de457c94d0222312d572a9e7eaf99"
server_key_iv: "64c8e4e3242bce4cae0e405f6bd572b2"
server_key_version: "encryptionkey@0"
I20260925 10:39:29.751266 27376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-2/wal/instance:
uuid: "87f1df5d2054405d8b6296d60029de6e"
format_stamp: "Formatted at 2026-09-25 10:39:29 on dist-test-slave-dxdd"
server_key: "c26de457c94d0222312d572a9e7eaf99"
server_key_iv: "64c8e4e3242bce4cae0e405f6bd572b2"
server_key_version: "encryptionkey@0"
I20260925 10:39:29.755875 27376 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 10:39:29.759140 27391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:29.760380 27376 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:39:29.760545 27376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-2/wal
uuid: "87f1df5d2054405d8b6296d60029de6e"
format_stamp: "Formatted at 2026-09-25 10:39:29 on dist-test-slave-dxdd"
server_key: "c26de457c94d0222312d572a9e7eaf99"
server_key_iv: "64c8e4e3242bce4cae0e405f6bd572b2"
server_key_version: "encryptionkey@0"
I20260925 10:39:29.760728 27376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:39:29.776093 27376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:39:29.780375 27376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:39:29.780653 27376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:39:29.781544 27376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:39:29.783018 27376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:39:29.783115 27376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:29.783272 27376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:39:29.783339 27376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:29.815871 27376 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.3:34031
I20260925 10:39:29.815917 27503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.3:34031 every 8 connection(s)
I20260925 10:39:29.817070 27376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
I20260925 10:39:29.827371 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 27376
I20260925 10:39:29.827600 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790332740062128-23952-0/minicluster-data/ts-2/wal/instance
I20260925 10:39:29.828141 23952 external_mini_cluster.cc:1428] Setting key e847ce7de36728081b077d00b45485b3
Sep 25 10:39:29 dist-test-slave-dxdd krb5kdc[27003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332769, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.3@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:39:29.836544 27239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:29.819568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:49547 (local address 127.23.100.62:38981)
0925 10:39:29.819748 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:29.819753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:29.820852 (+ 1099us) server_negotiation.cc:407] Connection header received
0925 10:39:29.822707 (+ 1855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:29.822713 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:29.822786 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:29.822877 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:29.824316 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:29.825012 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:29.826474 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:29.826847 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:29.826935 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:29.830430 (+ 3495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:29.830434 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:29.830437 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:29.830467 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:29.833434 (+ 2967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:29.834186 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:29.834191 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:29.834195 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:29.834379 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:29.834840 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:29.834845 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:29.834848 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:29.835145 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:29.835289 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:29.835988 (+ 699us) server_negotiation.cc:299] Negotiation successful
0925 10:39:29.836219 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 10:39:29.837536 27506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:29.820036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:38981 (local address 127.23.100.3:49547)
0925 10:39:29.820700 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:29.820743 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:29.822450 (+ 1707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:29.823076 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:29.823083 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:29.823586 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:29.824163 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:29.824180 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:29.825154 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:29.825157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:29.826295 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:29.826361 (+ 66us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:29.827018 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:29.827022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:29.827203 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:29.827932 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:29.827961 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:29.830170 (+ 2209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:29.833593 (+ 3423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:29.833603 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:29.833620 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:29.834055 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:29.834494 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:29.834499 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:29.834502 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:29.834718 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:29.835267 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:29.835274 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:29.835631 (+ 357us) client_negotiation.cc:769] Sending connection context
0925 10:39:29.835906 (+ 275us) client_negotiation.cc:240] Negotiation successful
0925 10:39:29.836426 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":267,"threads_started":1}
I20260925 10:39:29.838979 27504 heartbeater.cc:347] Connected to a master server at 127.23.100.62:38981
I20260925 10:39:29.839257 27504 heartbeater.cc:464] Registering TS with master...
I20260925 10:39:29.839926 27504 heartbeater.cc:511] Master 127.23.100.62:38981 requested a full tablet report, sending...
I20260925 10:39:29.841272 27048 ts_manager.cc:194] Registered new tserver with Master: 87f1df5d2054405d8b6296d60029de6e (127.23.100.3:34031)
I20260925 10:39:29.842453 27048 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.3@KRBTEST.COM'} at 127.23.100.3:49547
I20260925 10:39:29.843604 23952 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 10:39:29 dist-test-slave-dxdd krb5kdc[27003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332769, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 25 10:39:29 dist-test-slave-dxdd krb5kdc[27003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332769, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:39:29.890588 27237 heartbeater.cc:503] Master 127.23.100.62:38981 was elected leader, sending a full tablet report...
I20260925 10:39:29.891296 27239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:29.877738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33704 (local address 127.23.100.62:38981)
0925 10:39:29.877930 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:29.877937 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:29.878422 (+ 485us) server_negotiation.cc:407] Connection header received
0925 10:39:29.878654 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:29.878658 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:29.878718 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:29.878841 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:29.880166 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:29.880885 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:29.882072 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:29.882510 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:29.882667 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:29.886464 (+ 3797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:29.886469 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:29.886473 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:29.886504 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:29.888883 (+ 2379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:29.889441 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:29.889445 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:29.889447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:29.889568 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:29.889879 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:29.889882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:29.889884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:29.890177 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:29.890447 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:29.890763 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 10:39:29.890973 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260925 10:39:29.891782 27047 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:33704
I20260925 10:39:29.893441 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 27109
Sep 25 10:39:29 dist-test-slave-dxdd krb5kdc[27003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332769, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.23.100.1@KRBTEST.COM
I20260925 10:39:29.908349 27515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:29.894506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53312 (local address 127.23.100.1:45069)
0925 10:39:29.895227 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:29.895233 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:29.895256 (+ 23us) server_negotiation.cc:407] Connection header received
0925 10:39:29.895301 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:29.895307 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:29.895501 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:29.895618 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:29.896599 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:29.897550 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:29.898757 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:29.899323 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:29.899417 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:29.903267 (+ 3850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:29.903276 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:29.903289 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:29.903321 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:29.905878 (+ 2557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:29.906573 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:29.906580 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:29.906582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:29.906705 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:29.907033 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:29.907036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:29.907038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:29.907352 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:29.907563 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:29.907774 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 10:39:29.907987 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":436,"threads_started":1}
W20260925 10:39:29.909536 27232 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:53312
W20260925 10:39:29.910627 23952 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 27109: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:39:29.910802 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 27109
I20260925 10:39:29.926085 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 27243
Sep 25 10:39:29 dist-test-slave-dxdd krb5kdc[27003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332769, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.23.100.2@KRBTEST.COM
I20260925 10:39:29.940810 27516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:29.926901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43486 (local address 127.23.100.2:39465)
0925 10:39:29.927325 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:29.927331 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:29.927354 (+ 23us) server_negotiation.cc:407] Connection header received
0925 10:39:29.927408 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:29.927413 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:29.927648 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:29.927794 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:29.928810 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:29.929772 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:29.930991 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:29.931501 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:29.931597 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:29.935488 (+ 3891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:29.935500 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:29.935512 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:29.935573 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:29.938252 (+ 2679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:29.938972 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:29.938978 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:29.938980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:29.939103 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:29.939410 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:29.939413 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:29.939415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:29.939735 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:29.939950 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:29.940195 (+ 245us) server_negotiation.cc:299] Negotiation successful
0925 10:39:29.940401 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":210,"threads_started":1}
W20260925 10:39:29.942011 27366 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43486
W20260925 10:39:29.943113 23952 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 27243: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:39:29.943253 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 27243
I20260925 10:39:29.958926 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 27376
Sep 25 10:39:29 dist-test-slave-dxdd krb5kdc[27003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332769, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.23.100.3@KRBTEST.COM
I20260925 10:39:29.974490 27517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:29.959870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42282 (local address 127.23.100.3:34031)
0925 10:39:29.960535 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:29.960542 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:29.960570 (+ 28us) server_negotiation.cc:407] Connection header received
0925 10:39:29.960638 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:29.960644 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:29.960844 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:29.960980 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:29.962123 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:29.963190 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:29.964389 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:29.965019 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:29.965114 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:29.969179 (+ 4065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:29.969189 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:29.969202 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:29.969235 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:29.972215 (+ 2980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:29.972780 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:29.972786 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:29.972789 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:29.972905 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:29.973186 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:29.973189 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:29.973191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:29.973526 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:29.973744 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:29.973937 (+ 193us) server_negotiation.cc:299] Negotiation successful
0925 10:39:29.974114 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":355,"threads_started":1}
W20260925 10:39:29.975683 27499 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42282
W20260925 10:39:29.976675 23952 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 27376: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:39:29.976809 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 27376
I20260925 10:39:29.991871 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 27016
I20260925 10:39:29.992955 27078 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:39:30.093060 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 27016
2026-09-25T10:39:30Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3428 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:39:30 dist-test-slave-dxdd krb5kdc[27521](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:39:30 dist-test-slave-dxdd krb5kdc[27521](info): set up 2 sockets
Sep 25 10:39:30 dist-test-slave-dxdd krb5kdc[27521](info): commencing operation
krb5kdc: starting...
W20260925 10:39:31.170817 23952 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Sep 25 10:39:31 dist-test-slave-dxdd krb5kdc[27521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:39:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:39:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
I20260925 10:39:31.361951 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:46337
--webserver_interface=127.23.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:42045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.62:46337
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:39:31.668844 27534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:39:31.669133 27534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:39:31.669302 27534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:39:31.676223 27534 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:39:31.676357 27534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:39:31.676452 27534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:39:31.676548 27534 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:39:31.676620 27534 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:39:31.685968 27534 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:42045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.62:46337
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:46337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:39:31.687206 27534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:39:31.688449 27534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:39:31.700088 27539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:31.700572 27540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:31.700587 27542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:31.701112 27534 server_base.cc:1034] running on GCE node
I20260925 10:39:31.701927 27534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:39:31.703207 27534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:39:31.704397 27534 hybrid_clock.cc:648] HybridClock initialized: now 1790332771704342 us; error 45 us; skew 500 ppm
I20260925 10:39:31.704717 27534 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:39:31 dist-test-slave-dxdd krb5kdc[27521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332771, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:39:31.713013 27534 init.cc:377] Logged in from keytab as kudu/127.23.100.62@KRBTEST.COM (short username kudu)
I20260925 10:39:31.714425 27534 webserver.cc:460] Webserver started at http://127.23.100.62:43353/ using document root <none> and password file <none>
I20260925 10:39:31.715042 27534 fs_manager.cc:359] Metadata directory not provided
I20260925 10:39:31.715137 27534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:39:31.715376 27534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:39:31.718178 27534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/master-0/data/instance:
uuid: "29e2e2f22c9540eab520936f3c9234ec"
format_stamp: "Formatted at 2026-09-25 10:39:31 on dist-test-slave-dxdd"
server_key: "d4a49a5da458d838d325238c9b61db68"
server_key_iv: "ecf612b2995394633dcdf5669ca31089"
server_key_version: "encryptionkey@0"
I20260925 10:39:31.718952 27534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/master-0/wal/instance:
uuid: "29e2e2f22c9540eab520936f3c9234ec"
format_stamp: "Formatted at 2026-09-25 10:39:31 on dist-test-slave-dxdd"
server_key: "d4a49a5da458d838d325238c9b61db68"
server_key_iv: "ecf612b2995394633dcdf5669ca31089"
server_key_version: "encryptionkey@0"
I20260925 10:39:31.723663 27534 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260925 10:39:31.726783 27549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:31.727975 27534 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.002s
I20260925 10:39:31.728142 27534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/master-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/master-0/wal
uuid: "29e2e2f22c9540eab520936f3c9234ec"
format_stamp: "Formatted at 2026-09-25 10:39:31 on dist-test-slave-dxdd"
server_key: "d4a49a5da458d838d325238c9b61db68"
server_key_iv: "ecf612b2995394633dcdf5669ca31089"
server_key_version: "encryptionkey@0"
I20260925 10:39:31.728343 27534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:39:31.752113 27534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:39:31.756397 27534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:39:31.756673 27534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:39:31.774230 27534 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.62:46337
I20260925 10:39:31.774248 27600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.62:46337 every 8 connection(s)
I20260925 10:39:31.775516 27534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
I20260925 10:39:31.777956 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 27534
I20260925 10:39:31.778172 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/master-0/wal/instance
I20260925 10:39:31.778822 23952 external_mini_cluster.cc:1428] Setting key fe8eb0778e72f212f90f09a6b14bf142
I20260925 10:39:31.779240 27601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:39:31.792965 27601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29e2e2f22c9540eab520936f3c9234ec: Bootstrap starting.
Sep 25 10:39:31 dist-test-slave-dxdd krb5kdc[27521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:39:31.796808 27601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29e2e2f22c9540eab520936f3c9234ec: Neither blocks nor log segments found. Creating new log.
I20260925 10:39:31.798102 27601 log.cc:824] T 00000000000000000000000000000000 P 29e2e2f22c9540eab520936f3c9234ec: Log is configured to *not* fsync() on all Append() calls
I20260925 10:39:31.801368 27601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29e2e2f22c9540eab520936f3c9234ec: No bootstrap required, opened a new log
I20260925 10:39:31.802759 27604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:31.780720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51024 (local address 127.23.100.62:46337)
0925 10:39:31.781508 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:31.781523 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:31.781572 (+ 49us) server_negotiation.cc:407] Connection header received
0925 10:39:31.784002 (+ 2430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:31.784027 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:31.784525 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:31.784959 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:31.786198 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:31.788169 (+ 1971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:31.789379 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:31.790481 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:31.790671 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:31.794414 (+ 3743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:31.794426 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:31.794447 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:31.794507 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:31.798404 (+ 3897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:31.799217 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:31.799225 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:31.799231 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:31.799388 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:31.799795 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:31.799800 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:31.799803 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:31.800442 (+ 639us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:31.800730 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:31.801064 (+ 334us) server_negotiation.cc:299] Negotiation successful
0925 10:39:31.801473 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":333,"threads_started":1}
I20260925 10:39:31.809712 27601 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29e2e2f22c9540eab520936f3c9234ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29e2e2f22c9540eab520936f3c9234ec" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 46337 } }
I20260925 10:39:31.810019 27601 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29e2e2f22c9540eab520936f3c9234ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:39:31.810166 27601 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29e2e2f22c9540eab520936f3c9234ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29e2e2f22c9540eab520936f3c9234ec, State: Initialized, Role: FOLLOWER
I20260925 10:39:31.810843 27601 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29e2e2f22c9540eab520936f3c9234ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29e2e2f22c9540eab520936f3c9234ec" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 46337 } }
I20260925 10:39:31.811100 27601 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29e2e2f22c9540eab520936f3c9234ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:39:31.811241 27601 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29e2e2f22c9540eab520936f3c9234ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:39:31.811425 27601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29e2e2f22c9540eab520936f3c9234ec [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:39:31.813479 27601 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29e2e2f22c9540eab520936f3c9234ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29e2e2f22c9540eab520936f3c9234ec" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 46337 } }
I20260925 10:39:31.813951 27601 leader_election.cc:302] T 00000000000000000000000000000000 P 29e2e2f22c9540eab520936f3c9234ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29e2e2f22c9540eab520936f3c9234ec; no voters:
I20260925 10:39:31.814568 27601 leader_election.cc:288] T 00000000000000000000000000000000 P 29e2e2f22c9540eab520936f3c9234ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:39:31.814703 27606 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29e2e2f22c9540eab520936f3c9234ec [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:39:31.815376 27606 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29e2e2f22c9540eab520936f3c9234ec [term 1 LEADER]: Becoming Leader. State: Replica: 29e2e2f22c9540eab520936f3c9234ec, State: Running, Role: LEADER
I20260925 10:39:31.815954 27606 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29e2e2f22c9540eab520936f3c9234ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29e2e2f22c9540eab520936f3c9234ec" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 46337 } }
I20260925 10:39:31.816510 27601 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29e2e2f22c9540eab520936f3c9234ec [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:39:31.818622 27607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29e2e2f22c9540eab520936f3c9234ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29e2e2f22c9540eab520936f3c9234ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29e2e2f22c9540eab520936f3c9234ec" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 46337 } } }
I20260925 10:39:31.818977 27607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29e2e2f22c9540eab520936f3c9234ec [sys.catalog]: This master's current role is: LEADER
I20260925 10:39:31.819447 27608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29e2e2f22c9540eab520936f3c9234ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29e2e2f22c9540eab520936f3c9234ec. Latest consensus state: current_term: 1 leader_uuid: "29e2e2f22c9540eab520936f3c9234ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29e2e2f22c9540eab520936f3c9234ec" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 46337 } } }
I20260925 10:39:31.825850 27608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29e2e2f22c9540eab520936f3c9234ec [sys.catalog]: This master's current role is: LEADER
I20260925 10:39:31.827234 27612 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:39:31.831358 27612 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:39:31.839342 27612 catalog_manager.cc:1347] Generated new cluster ID: ce32b82260b64b97bba81074b0577501
I20260925 10:39:31.839452 27612 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:39:31.858971 27612 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:39:31.860185 27612 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:39:31.865994 27612 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29e2e2f22c9540eab520936f3c9234ec: Generated new TSK 0
I20260925 10:39:31.866779 27612 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab.
Entry for principal kudu/127.23.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab.
I20260925 10:39:31.919646 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.1:0
--local_ip_for_outbound_sockets=127.23.100.1
--webserver_interface=127.23.100.1
--webserver_port=0
--tserver_master_addrs=127.23.100.62:46337
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab
--principal=kudu/127.23.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:42045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:39:32.231571 27627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:39:32.231835 27627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:39:32.231966 27627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:39:32.238865 27627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:39:32.238977 27627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:39:32.239143 27627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.1
I20260925 10:39:32.248906 27627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:42045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab
--principal=kudu/127.23.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.1
--webserver_port=0
--tserver_master_addrs=127.23.100.62:46337
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.1
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:39:32.250065 27627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:39:32.251355 27627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:39:32.263773 27632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:32.263831 27633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:32.263983 27635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:32.264696 27627 server_base.cc:1034] running on GCE node
I20260925 10:39:32.265239 27627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:39:32.266076 27627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:39:32.267288 27627 hybrid_clock.cc:648] HybridClock initialized: now 1790332772267229 us; error 71 us; skew 500 ppm
I20260925 10:39:32.267578 27627 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:39:32 dist-test-slave-dxdd krb5kdc[27521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332772, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:39:32.270545 27627 init.cc:377] Logged in from keytab as kudu/127.23.100.1@KRBTEST.COM (short username kudu)
I20260925 10:39:32.271905 27627 webserver.cc:460] Webserver started at http://127.23.100.1:38761/ using document root <none> and password file <none>
I20260925 10:39:32.272539 27627 fs_manager.cc:359] Metadata directory not provided
I20260925 10:39:32.272688 27627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:39:32.272986 27627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:39:32.275761 27627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/ts-0/data/instance:
uuid: "9cd8164c107f43a68193efd0ccb53c74"
format_stamp: "Formatted at 2026-09-25 10:39:32 on dist-test-slave-dxdd"
server_key: "2cfc3f0e18e70f3b0d9936a71c56d26d"
server_key_iv: "aa0230d3fe4457454689b6c20e6a85b8"
server_key_version: "encryptionkey@0"
I20260925 10:39:32.276515 27627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/ts-0/wal/instance:
uuid: "9cd8164c107f43a68193efd0ccb53c74"
format_stamp: "Formatted at 2026-09-25 10:39:32 on dist-test-slave-dxdd"
server_key: "2cfc3f0e18e70f3b0d9936a71c56d26d"
server_key_iv: "aa0230d3fe4457454689b6c20e6a85b8"
server_key_version: "encryptionkey@0"
I20260925 10:39:32.281493 27627 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260925 10:39:32.284711 27642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:32.285921 27627 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 10:39:32.286087 27627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/ts-0/wal
uuid: "9cd8164c107f43a68193efd0ccb53c74"
format_stamp: "Formatted at 2026-09-25 10:39:32 on dist-test-slave-dxdd"
server_key: "2cfc3f0e18e70f3b0d9936a71c56d26d"
server_key_iv: "aa0230d3fe4457454689b6c20e6a85b8"
server_key_version: "encryptionkey@0"
I20260925 10:39:32.286280 27627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:39:32.303755 27627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:39:32.307852 27627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:39:32.308115 27627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:39:32.308974 27627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:39:32.310243 27627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:39:32.310389 27627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:32.310539 27627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:39:32.310609 27627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:32.342751 27627 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.1:35587
I20260925 10:39:32.342794 27754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.1:35587 every 8 connection(s)
I20260925 10:39:32.344074 27627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
I20260925 10:39:32.348531 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 27627
I20260925 10:39:32.348755 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/ts-0/wal/instance
I20260925 10:39:32.349570 23952 external_mini_cluster.cc:1428] Setting key 06d6152432cd251127b31c8d367cf847
Sep 25 10:39:32 dist-test-slave-dxdd krb5kdc[27521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332772, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.1@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:39:32.366788 27757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:32.346535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:45827 (local address 127.23.100.62:46337)
0925 10:39:32.346984 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:32.346989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:32.347872 (+ 883us) server_negotiation.cc:407] Connection header received
0925 10:39:32.350509 (+ 2637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:32.350514 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:32.350608 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:32.350728 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:32.352446 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:32.353118 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:32.355581 (+ 2463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:32.355992 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:32.356083 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:32.360007 (+ 3924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:32.360015 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:32.360023 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:32.360067 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:32.363538 (+ 3471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:32.364228 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:32.364236 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:32.364239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:32.364376 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:32.364793 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:32.364800 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:32.364806 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:32.365215 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:32.365364 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:32.366022 (+ 658us) server_negotiation.cc:299] Negotiation successful
0925 10:39:32.366314 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":238,"threads_started":1}
I20260925 10:39:32.367533 27758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:32.347031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:46337 (local address 127.23.100.1:45827)
0925 10:39:32.347710 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:32.347754 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:32.350173 (+ 2419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:32.350967 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:32.350975 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:32.351479 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:32.352244 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:32.352288 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:32.353275 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:32.353283 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:32.355440 (+ 2157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:32.355457 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:32.356215 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:32.356224 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:32.356541 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:32.357246 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:32.357283 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:32.359788 (+ 2505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:32.363694 (+ 3906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:32.363701 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:32.363713 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:32.364096 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:32.364475 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:32.364478 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:32.364481 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:32.364674 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:32.365342 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:32.365348 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:32.365643 (+ 295us) client_negotiation.cc:769] Sending connection context
0925 10:39:32.365870 (+ 227us) client_negotiation.cc:240] Negotiation successful
0925 10:39:32.366430 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":259,"threads_started":1}
I20260925 10:39:32.369171 27755 heartbeater.cc:347] Connected to a master server at 127.23.100.62:46337
I20260925 10:39:32.369495 27755 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.23.100.2@KRBTEST.COM; defaulting to no policy
I20260925 10:39:32.370186 27755 heartbeater.cc:511] Master 127.23.100.62:46337 requested a full tablet report, sending...
I20260925 10:39:32.372298 27566 ts_manager.cc:194] Registered new tserver with Master: 9cd8164c107f43a68193efd0ccb53c74 (127.23.100.1:35587)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.2@KRBTEST.COM" created.
I20260925 10:39:32.373875 27566 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.1@KRBTEST.COM'} at 127.23.100.1:45827
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab.
Entry for principal kudu/127.23.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab.
I20260925 10:39:32.395773 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.2:0
--local_ip_for_outbound_sockets=127.23.100.2
--webserver_interface=127.23.100.2
--webserver_port=0
--tserver_master_addrs=127.23.100.62:46337
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab
--principal=kudu/127.23.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:42045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:39:32.704531 27761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:39:32.704838 27761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:39:32.704988 27761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:39:32.712073 27761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:39:32.712174 27761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:39:32.712385 27761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.2
I20260925 10:39:32.721575 27761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:42045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab
--principal=kudu/127.23.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.2
--webserver_port=0
--tserver_master_addrs=127.23.100.62:46337
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.2
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:39:32.722851 27761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:39:32.724244 27761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:39:32.736941 27766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:32.737021 27767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:32.736980 27769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:32.737772 27761 server_base.cc:1034] running on GCE node
I20260925 10:39:32.738379 27761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:39:32.739269 27761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:39:32.740487 27761 hybrid_clock.cc:648] HybridClock initialized: now 1790332772740444 us; error 39 us; skew 500 ppm
I20260925 10:39:32.740790 27761 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:39:32 dist-test-slave-dxdd krb5kdc[27521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332772, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:39:32.743752 27761 init.cc:377] Logged in from keytab as kudu/127.23.100.2@KRBTEST.COM (short username kudu)
I20260925 10:39:32.745085 27761 webserver.cc:460] Webserver started at http://127.23.100.2:35707/ using document root <none> and password file <none>
I20260925 10:39:32.745678 27761 fs_manager.cc:359] Metadata directory not provided
I20260925 10:39:32.745775 27761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:39:32.746019 27761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:39:32.748750 27761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/ts-1/data/instance:
uuid: "8394abb32bac4e09aa137743c5064723"
format_stamp: "Formatted at 2026-09-25 10:39:32 on dist-test-slave-dxdd"
server_key: "519c09441099ea32dc47240d8aa602bc"
server_key_iv: "dde5532aed4a6777c9f37c12d6a8c21d"
server_key_version: "encryptionkey@0"
I20260925 10:39:32.749449 27761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/ts-1/wal/instance:
uuid: "8394abb32bac4e09aa137743c5064723"
format_stamp: "Formatted at 2026-09-25 10:39:32 on dist-test-slave-dxdd"
server_key: "519c09441099ea32dc47240d8aa602bc"
server_key_iv: "dde5532aed4a6777c9f37c12d6a8c21d"
server_key_version: "encryptionkey@0"
I20260925 10:39:32.754190 27761 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260925 10:39:32.757489 27776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:32.758810 27761 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:39:32.758961 27761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/ts-1/wal
uuid: "8394abb32bac4e09aa137743c5064723"
format_stamp: "Formatted at 2026-09-25 10:39:32 on dist-test-slave-dxdd"
server_key: "519c09441099ea32dc47240d8aa602bc"
server_key_iv: "dde5532aed4a6777c9f37c12d6a8c21d"
server_key_version: "encryptionkey@0"
I20260925 10:39:32.759161 27761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:39:32.773986 27761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:39:32.777899 27761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:39:32.778168 27761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:39:32.779182 27761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:39:32.780439 27761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:39:32.780512 27761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:32.780652 27761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:39:32.780701 27761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:32.814255 27761 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.2:36797
I20260925 10:39:32.814345 27888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.2:36797 every 8 connection(s)
I20260925 10:39:32.815547 27761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
I20260925 10:39:32.822263 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 27761
I20260925 10:39:32.822479 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/ts-1/wal/instance
I20260925 10:39:32.823004 23952 external_mini_cluster.cc:1428] Setting key 7bb6236e3ab3c018f66d0e27a08c2896
Sep 25 10:39:32 dist-test-slave-dxdd krb5kdc[27521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332772, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.2@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:39:32.836768 27757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:32.818081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:54811 (local address 127.23.100.62:46337)
0925 10:39:32.818259 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:32.818263 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:32.819455 (+ 1192us) server_negotiation.cc:407] Connection header received
0925 10:39:32.821173 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:32.821177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:32.821266 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:32.821354 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:32.823202 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:32.824173 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:32.826063 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:32.826505 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:32.826599 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:32.830560 (+ 3961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:32.830565 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:32.830568 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:32.830597 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:32.833545 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:32.834466 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:32.834470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:32.834472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:32.834594 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:32.835065 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:32.835068 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:32.835070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:32.835313 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:32.835417 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:32.836237 (+ 820us) server_negotiation.cc:299] Negotiation successful
0925 10:39:32.836403 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 10:39:32.838145 27891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:32.818618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:46337 (local address 127.23.100.2:54811)
0925 10:39:32.819290 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:32.819334 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:32.820930 (+ 1596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:32.821550 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:32.821558 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:32.822197 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:32.823036 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:32.823062 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:32.824312 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:32.824317 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:32.825920 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:32.825937 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:32.826730 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:32.826740 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:32.827062 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:32.827776 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:32.827810 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:32.830353 (+ 2543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:32.833676 (+ 3323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:32.833687 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:32.833705 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:32.834353 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:32.834707 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:32.834715 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:32.834722 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:32.834956 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:32.835433 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:32.835443 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:32.835881 (+ 438us) client_negotiation.cc:769] Sending connection context
0925 10:39:32.836164 (+ 283us) client_negotiation.cc:240] Negotiation successful
0925 10:39:32.836728 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":258,"threads_started":1}
I20260925 10:39:32.839579 27889 heartbeater.cc:347] Connected to a master server at 127.23.100.62:46337
I20260925 10:39:32.839875 27889 heartbeater.cc:464] Registering TS with master...
I20260925 10:39:32.840545 27889 heartbeater.cc:511] Master 127.23.100.62:46337 requested a full tablet report, sending...
I20260925 10:39:32.841889 27566 ts_manager.cc:194] Registered new tserver with Master: 8394abb32bac4e09aa137743c5064723 (127.23.100.2:36797)
I20260925 10:39:32.842731 27566 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.2@KRBTEST.COM'} at 127.23.100.2:54811
WARNING: no policy specified for kudu/127.23.100.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab.
Entry for principal kudu/127.23.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab.
I20260925 10:39:32.867280 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.3:0
--local_ip_for_outbound_sockets=127.23.100.3
--webserver_interface=127.23.100.3
--webserver_port=0
--tserver_master_addrs=127.23.100.62:46337
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab
--principal=kudu/127.23.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:42045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:39:33.175472 27894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:39:33.175746 27894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:39:33.175884 27894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:39:33.182904 27894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:39:33.183040 27894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:39:33.183233 27894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.3
I20260925 10:39:33.192761 27894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:42045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab
--principal=kudu/127.23.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.3
--webserver_port=0
--tserver_master_addrs=127.23.100.62:46337
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.3
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:39:33.194015 27894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:39:33.195309 27894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:39:33.208169 27899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:33.208103 27902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:33.208139 27900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:33.209230 27894 server_base.cc:1034] running on GCE node
I20260925 10:39:33.209820 27894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:39:33.210776 27894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:39:33.211961 27894 hybrid_clock.cc:648] HybridClock initialized: now 1790332773211923 us; error 35 us; skew 500 ppm
I20260925 10:39:33.212271 27894 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:39:33 dist-test-slave-dxdd krb5kdc[27521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332773, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:39:33.215301 27894 init.cc:377] Logged in from keytab as kudu/127.23.100.3@KRBTEST.COM (short username kudu)
I20260925 10:39:33.216789 27894 webserver.cc:460] Webserver started at http://127.23.100.3:45863/ using document root <none> and password file <none>
I20260925 10:39:33.217439 27894 fs_manager.cc:359] Metadata directory not provided
I20260925 10:39:33.217610 27894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:39:33.217939 27894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:39:33.220786 27894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/ts-2/data/instance:
uuid: "1db0b13e728a437da59baf3b544c8d17"
format_stamp: "Formatted at 2026-09-25 10:39:33 on dist-test-slave-dxdd"
server_key: "eac6bcd2253f8ae7f8a5eba480153e2c"
server_key_iv: "cb1c81873e619ca2e0d2a3e2d46c21b1"
server_key_version: "encryptionkey@0"
I20260925 10:39:33.221508 27894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/ts-2/wal/instance:
uuid: "1db0b13e728a437da59baf3b544c8d17"
format_stamp: "Formatted at 2026-09-25 10:39:33 on dist-test-slave-dxdd"
server_key: "eac6bcd2253f8ae7f8a5eba480153e2c"
server_key_iv: "cb1c81873e619ca2e0d2a3e2d46c21b1"
server_key_version: "encryptionkey@0"
I20260925 10:39:33.226176 27894 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 10:39:33.230036 27909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:33.231762 27894 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 10:39:33.231936 27894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/ts-2/wal
uuid: "1db0b13e728a437da59baf3b544c8d17"
format_stamp: "Formatted at 2026-09-25 10:39:33 on dist-test-slave-dxdd"
server_key: "eac6bcd2253f8ae7f8a5eba480153e2c"
server_key_iv: "cb1c81873e619ca2e0d2a3e2d46c21b1"
server_key_version: "encryptionkey@0"
I20260925 10:39:33.232187 27894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:39:33.249753 27894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:39:33.253372 27894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:39:33.253656 27894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:39:33.254552 27894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:39:33.255789 27894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:39:33.255858 27894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:33.256017 27894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:39:33.256078 27894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:33.288172 27894 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.3:37429
I20260925 10:39:33.288209 28021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.3:37429 every 8 connection(s)
I20260925 10:39:33.289361 27894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
I20260925 10:39:33.293602 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 27894
I20260925 10:39:33.293800 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790332740062128-23952-0/minicluster-data/ts-2/wal/instance
I20260925 10:39:33.294200 23952 external_mini_cluster.cc:1428] Setting key c0ec96f80f15a0cdd28fc18eaa3f1406
Sep 25 10:39:33 dist-test-slave-dxdd krb5kdc[27521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332773, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.3@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:39:33.308096 27757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:33.291856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:60669 (local address 127.23.100.62:46337)
0925 10:39:33.292020 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:33.292024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:33.293066 (+ 1042us) server_negotiation.cc:407] Connection header received
0925 10:39:33.295090 (+ 2024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:33.295095 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:33.295190 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:33.295296 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:33.296708 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:33.297536 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:33.299123 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:33.299531 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:33.299656 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:33.302835 (+ 3179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:33.302841 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:33.302844 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:33.302873 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:33.305178 (+ 2305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:33.305867 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:33.305874 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:33.305900 (+ 26us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:33.306007 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:33.306459 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:33.306464 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:33.306469 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:33.306721 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:33.306872 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:33.307510 (+ 638us) server_negotiation.cc:299] Negotiation successful
0925 10:39:33.307764 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 10:39:33.309190 28024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:33.292252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:46337 (local address 127.23.100.3:60669)
0925 10:39:33.292899 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:33.292933 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:33.294802 (+ 1869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:33.295522 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:33.295529 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:33.296015 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:33.296541 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:33.296556 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:33.297719 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:33.297723 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:33.298986 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:33.298997 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:33.299762 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:33.299769 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:33.299991 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:33.300557 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:33.300581 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:33.302633 (+ 2052us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:33.305315 (+ 2682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:33.305323 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:33.305337 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:33.305727 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:33.306115 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:33.306120 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:33.306124 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:33.306295 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:33.306845 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:33.306853 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:33.307171 (+ 318us) client_negotiation.cc:769] Sending connection context
0925 10:39:33.307412 (+ 241us) client_negotiation.cc:240] Negotiation successful
0925 10:39:33.307886 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":264,"threads_started":1}
I20260925 10:39:33.310421 28022 heartbeater.cc:347] Connected to a master server at 127.23.100.62:46337
I20260925 10:39:33.310711 28022 heartbeater.cc:464] Registering TS with master...
I20260925 10:39:33.311380 28022 heartbeater.cc:511] Master 127.23.100.62:46337 requested a full tablet report, sending...
I20260925 10:39:33.312639 27566 ts_manager.cc:194] Registered new tserver with Master: 1db0b13e728a437da59baf3b544c8d17 (127.23.100.3:37429)
I20260925 10:39:33.313390 27566 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.3@KRBTEST.COM'} at 127.23.100.3:60669
I20260925 10:39:33.320923 23952 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 10:39:33 dist-test-slave-dxdd krb5kdc[27521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 10:39:33 dist-test-slave-dxdd krb5kdc[27521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.1@KRBTEST.COM
I20260925 10:39:33.367329 28029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:33.353549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37550 (local address 127.23.100.1:35587)
0925 10:39:33.354098 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:33.354105 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:33.354129 (+ 24us) server_negotiation.cc:407] Connection header received
0925 10:39:33.354190 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:33.354206 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:33.354475 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:33.354601 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:33.355908 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:33.356911 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:33.358072 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:33.358659 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:33.358753 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:33.362254 (+ 3501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:33.362263 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:33.362275 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:33.362359 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:33.364909 (+ 2550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:33.365484 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:33.365490 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:33.365492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:33.365613 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:33.365922 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:33.365925 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:33.365927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:33.366243 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:33.366538 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:33.366754 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 10:39:33.366935 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":294,"threads_started":1}
I20260925 10:39:33.377352 27755 heartbeater.cc:503] Master 127.23.100.62:46337 was elected leader, sending a full tablet report...
Sep 25 10:39:33 dist-test-slave-dxdd krb5kdc[27521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:39:33.384191 27757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:33.371269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51026 (local address 127.23.100.62:46337)
0925 10:39:33.371468 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:33.371474 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:33.371620 (+ 146us) server_negotiation.cc:407] Connection header received
0925 10:39:33.371851 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:33.371855 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:33.371916 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:33.372014 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:33.373173 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:33.373875 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:33.375150 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:33.375586 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:33.375719 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:33.379827 (+ 4108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:33.379832 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:33.379835 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:33.379864 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:33.382100 (+ 2236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:33.382721 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:33.382725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:33.382727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:33.382845 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:33.383133 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:33.383136 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:33.383138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:33.383423 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:33.383607 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:33.383816 (+ 209us) server_negotiation.cc:299] Negotiation successful
0925 10:39:33.383960 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260925 10:39:33.384785 27566 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51026
I20260925 10:39:33.385818 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 27627
I20260925 10:39:33.398010 28029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:33.386711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37558 (local address 127.23.100.1:35587)
0925 10:39:33.386891 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:33.386898 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:33.387168 (+ 270us) server_negotiation.cc:407] Connection header received
0925 10:39:33.387346 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:33.387352 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:33.387419 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:33.387517 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:33.388637 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:33.389326 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:33.390657 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:33.391058 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:33.391206 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:33.393266 (+ 2060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:33.393277 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:33.393282 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:33.393310 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:33.395630 (+ 2320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:33.396236 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:33.396243 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:33.396247 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:33.396365 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:33.396706 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:33.396713 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:33.396718 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:33.396949 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:33.397182 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:33.397472 (+ 290us) server_negotiation.cc:299] Negotiation successful
0925 10:39:33.397709 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 10:39:33.398576 27750 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:39:33.480146 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 27627
I20260925 10:39:33.495956 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 27761
Sep 25 10:39:33 dist-test-slave-dxdd krb5kdc[27521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.2@KRBTEST.COM
I20260925 10:39:33.511513 28034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:33.496811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58250 (local address 127.23.100.2:36797)
0925 10:39:33.497384 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:33.497393 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:33.497423 (+ 30us) server_negotiation.cc:407] Connection header received
0925 10:39:33.497493 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:33.497501 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:33.497772 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:33.497941 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:33.499207 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:33.500174 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:33.501473 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:33.502010 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:33.502155 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:33.505980 (+ 3825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:33.505990 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:33.506003 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:33.506033 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:33.508624 (+ 2591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:33.509343 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:33.509351 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:33.509353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:33.509479 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:33.509864 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:33.509868 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:33.509870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:33.510172 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:33.510497 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:33.510901 (+ 404us) server_negotiation.cc:299] Negotiation successful
0925 10:39:33.511165 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":231,"threads_started":1}
I20260925 10:39:33.512742 27884 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:39:33.592936 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 27761
I20260925 10:39:33.608906 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 27894
Sep 25 10:39:33 dist-test-slave-dxdd krb5kdc[27521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.3@KRBTEST.COM
I20260925 10:39:33.624517 28036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:33.609955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52296 (local address 127.23.100.3:37429)
0925 10:39:33.610587 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:33.610594 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:33.610622 (+ 28us) server_negotiation.cc:407] Connection header received
0925 10:39:33.610681 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:33.610688 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:33.610900 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:33.611039 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:33.612213 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:33.613216 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:33.614563 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:33.615118 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:33.615214 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:33.619299 (+ 4085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:33.619310 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:33.619324 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:33.619356 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:33.621909 (+ 2553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:33.622565 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:33.622573 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:33.622575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:33.622691 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:33.623019 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:33.623021 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:33.623023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:33.623409 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:33.623691 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:33.623974 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 10:39:33.624176 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":270,"threads_started":1}
I20260925 10:39:33.625973 28017 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:39:33.707500 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 27894
I20260925 10:39:33.723407 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 27534
I20260925 10:39:33.724603 27596 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:39:33.826287 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 27534
2026-09-25T10:39:33Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3734 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:39:33 dist-test-slave-dxdd krb5kdc[28041](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:39:33 dist-test-slave-dxdd krb5kdc[28041](info): set up 2 sockets
Sep 25 10:39:33 dist-test-slave-dxdd krb5kdc[28041](info): commencing operation
krb5kdc: starting...
W20260925 10:39:34.914863 23952 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Sep 25 10:39:35 dist-test-slave-dxdd krb5kdc[28041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:39:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:39:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
I20260925 10:39:35.113739 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:43879
--webserver_interface=127.23.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:39413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.62:43879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:39:35.426200 28054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:39:35.426471 28054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:39:35.426625 28054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:39:35.433736 28054 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:39:35.433845 28054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:39:35.433897 28054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:39:35.433991 28054 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:39:35.434082 28054 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:39:35.443769 28054 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:39413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.62:43879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:43879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:39:35.444998 28054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:39:35.446276 28054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:39:35.457756 28059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:35.457715 28060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:35.457803 28062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:35.463641 28054 server_base.cc:1034] running on GCE node
I20260925 10:39:35.464393 28054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:39:35.465704 28054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:39:35.466939 28054 hybrid_clock.cc:648] HybridClock initialized: now 1790332775466884 us; error 44 us; skew 500 ppm
I20260925 10:39:35.467310 28054 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:39:35 dist-test-slave-dxdd krb5kdc[28041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332775, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:39:35.470253 28054 init.cc:377] Logged in from keytab as kudu/127.23.100.62@KRBTEST.COM (short username kudu)
I20260925 10:39:35.471501 28054 fs_manager.cc:359] Metadata directory not provided
I20260925 10:39:35.471637 28054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:39:35.471962 28054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:39:35.474834 28054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/master-0/data/instance:
uuid: "91202005ee564d6aa380909adebda070"
format_stamp: "Formatted at 2026-09-25 10:39:35 on dist-test-slave-dxdd"
server_key: "3715871b361e90ac5320ad22cd4aa6f2"
server_key_iv: "af9ca66a59b5f2465736a67afb45a502"
server_key_version: "encryptionkey@0"
I20260925 10:39:35.475623 28054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/master-0/wal/instance:
uuid: "91202005ee564d6aa380909adebda070"
format_stamp: "Formatted at 2026-09-25 10:39:35 on dist-test-slave-dxdd"
server_key: "3715871b361e90ac5320ad22cd4aa6f2"
server_key_iv: "af9ca66a59b5f2465736a67afb45a502"
server_key_version: "encryptionkey@0"
I20260925 10:39:35.480526 28054 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 10:39:35.483728 28068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:35.484953 28054 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 10:39:35.485142 28054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/master-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/master-0/wal
uuid: "91202005ee564d6aa380909adebda070"
format_stamp: "Formatted at 2026-09-25 10:39:35 on dist-test-slave-dxdd"
server_key: "3715871b361e90ac5320ad22cd4aa6f2"
server_key_iv: "af9ca66a59b5f2465736a67afb45a502"
server_key_version: "encryptionkey@0"
I20260925 10:39:35.485358 28054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:39:35.503520 28054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:39:35.507215 28054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:39:35.507477 28054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:39:35.524852 28054 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.62:43879
I20260925 10:39:35.524890 28119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.62:43879 every 8 connection(s)
I20260925 10:39:35.525745 28054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
I20260925 10:39:35.529153 28120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:39:35.532253 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 28054
I20260925 10:39:35.532482 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/master-0/wal/instance
I20260925 10:39:35.533061 23952 external_mini_cluster.cc:1428] Setting key 1d3fad311c34ba86790a8708e7608cd8
I20260925 10:39:35.541352 28120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91202005ee564d6aa380909adebda070: Bootstrap starting.
I20260925 10:39:35.545083 28120 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91202005ee564d6aa380909adebda070: Neither blocks nor log segments found. Creating new log.
I20260925 10:39:35.546336 28120 log.cc:824] T 00000000000000000000000000000000 P 91202005ee564d6aa380909adebda070: Log is configured to *not* fsync() on all Append() calls
Sep 25 10:39:35 dist-test-slave-dxdd krb5kdc[28041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:39:35.549283 28120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91202005ee564d6aa380909adebda070: No bootstrap required, opened a new log
I20260925 10:39:35.556658 28123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:35.535009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41186 (local address 127.23.100.62:43879)
0925 10:39:35.535794 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:35.535805 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:35.535844 (+ 39us) server_negotiation.cc:407] Connection header received
0925 10:39:35.537323 (+ 1479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:35.537342 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:35.537696 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:35.538046 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:35.539837 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:35.541074 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:35.542888 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:35.543553 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:35.543681 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:35.548448 (+ 4767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:35.548458 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:35.548478 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:35.548523 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:35.552453 (+ 3930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:35.553094 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:35.553102 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:35.553109 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:35.553302 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:35.553629 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:35.553634 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:35.553637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:35.554283 (+ 646us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:35.554591 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:35.554922 (+ 331us) server_negotiation.cc:299] Negotiation successful
0925 10:39:35.555339 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":393,"threads_started":1}
I20260925 10:39:35.559664 28120 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91202005ee564d6aa380909adebda070 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91202005ee564d6aa380909adebda070" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 43879 } }
I20260925 10:39:35.560027 28120 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91202005ee564d6aa380909adebda070 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:39:35.560109 28120 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91202005ee564d6aa380909adebda070 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91202005ee564d6aa380909adebda070, State: Initialized, Role: FOLLOWER
I20260925 10:39:35.560557 28120 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91202005ee564d6aa380909adebda070 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91202005ee564d6aa380909adebda070" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 43879 } }
I20260925 10:39:35.560747 28120 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91202005ee564d6aa380909adebda070 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:39:35.560834 28120 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91202005ee564d6aa380909adebda070 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:39:35.560947 28120 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91202005ee564d6aa380909adebda070 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:39:35.562969 28120 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91202005ee564d6aa380909adebda070 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91202005ee564d6aa380909adebda070" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 43879 } }
I20260925 10:39:35.563491 28120 leader_election.cc:302] T 00000000000000000000000000000000 P 91202005ee564d6aa380909adebda070 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91202005ee564d6aa380909adebda070; no voters:
I20260925 10:39:35.564077 28120 leader_election.cc:288] T 00000000000000000000000000000000 P 91202005ee564d6aa380909adebda070 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:39:35.564221 28125 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91202005ee564d6aa380909adebda070 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:39:35.565055 28125 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91202005ee564d6aa380909adebda070 [term 1 LEADER]: Becoming Leader. State: Replica: 91202005ee564d6aa380909adebda070, State: Running, Role: LEADER
I20260925 10:39:35.565582 28125 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91202005ee564d6aa380909adebda070 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91202005ee564d6aa380909adebda070" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 43879 } }
I20260925 10:39:35.566154 28120 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91202005ee564d6aa380909adebda070 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:39:35.568578 28126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91202005ee564d6aa380909adebda070 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91202005ee564d6aa380909adebda070" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91202005ee564d6aa380909adebda070" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 43879 } } }
I20260925 10:39:35.568609 28127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91202005ee564d6aa380909adebda070 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91202005ee564d6aa380909adebda070. Latest consensus state: current_term: 1 leader_uuid: "91202005ee564d6aa380909adebda070" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91202005ee564d6aa380909adebda070" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 43879 } } }
I20260925 10:39:35.569023 28127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91202005ee564d6aa380909adebda070 [sys.catalog]: This master's current role is: LEADER
I20260925 10:39:35.568915 28126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91202005ee564d6aa380909adebda070 [sys.catalog]: This master's current role is: LEADER
I20260925 10:39:35.570760 28134 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:39:35.575163 28134 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:39:35.583730 28134 catalog_manager.cc:1347] Generated new cluster ID: 65f49d13559d4ab29721454915db6d5f
I20260925 10:39:35.583870 28134 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:39:35.593495 28134 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:39:35.594828 28134 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:39:35.601876 28134 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91202005ee564d6aa380909adebda070: Generated new TSK 0
I20260925 10:39:35.602813 28134 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab.
Entry for principal kudu/127.23.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab.
I20260925 10:39:35.653836 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.1:0
--local_ip_for_outbound_sockets=127.23.100.1
--webserver_interface=127.23.100.1
--webserver_port=0
--tserver_master_addrs=127.23.100.62:43879
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab
--principal=kudu/127.23.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:39413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:39:35.976415 28146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:39:35.976706 28146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:39:35.976874 28146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:39:35.984099 28146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:39:35.984227 28146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:39:35.984411 28146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.1
I20260925 10:39:35.994460 28146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:39413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab
--principal=kudu/127.23.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.1
--webserver_port=0
--tserver_master_addrs=127.23.100.62:43879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.1
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:39:35.995764 28146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:39:35.997126 28146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:39:36.010345 28151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:36.010412 28152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:36.010435 28154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:36.011101 28146 server_base.cc:1034] running on GCE node
I20260925 10:39:36.011699 28146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:39:36.012616 28146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:39:36.013811 28146 hybrid_clock.cc:648] HybridClock initialized: now 1790332776013768 us; error 33 us; skew 500 ppm
I20260925 10:39:36.014133 28146 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:39:36 dist-test-slave-dxdd krb5kdc[28041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332776, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:39:36.017279 28146 init.cc:377] Logged in from keytab as kudu/127.23.100.1@KRBTEST.COM (short username kudu)
I20260925 10:39:36.018568 28146 fs_manager.cc:359] Metadata directory not provided
I20260925 10:39:36.018705 28146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:39:36.019032 28146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:39:36.021977 28146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/ts-0/data/instance:
uuid: "4f553d0fe89a466cb64c44f36098674f"
format_stamp: "Formatted at 2026-09-25 10:39:36 on dist-test-slave-dxdd"
server_key: "1ecdba22133f899a9c7eb1abf1985e80"
server_key_iv: "7bd27525d2090d642bc2b28940a6f854"
server_key_version: "encryptionkey@0"
I20260925 10:39:36.022804 28146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/ts-0/wal/instance:
uuid: "4f553d0fe89a466cb64c44f36098674f"
format_stamp: "Formatted at 2026-09-25 10:39:36 on dist-test-slave-dxdd"
server_key: "1ecdba22133f899a9c7eb1abf1985e80"
server_key_iv: "7bd27525d2090d642bc2b28940a6f854"
server_key_version: "encryptionkey@0"
I20260925 10:39:36.027688 28146 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 10:39:36.030975 28160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:36.032326 28146 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260925 10:39:36.032500 28146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/ts-0/wal
uuid: "4f553d0fe89a466cb64c44f36098674f"
format_stamp: "Formatted at 2026-09-25 10:39:36 on dist-test-slave-dxdd"
server_key: "1ecdba22133f899a9c7eb1abf1985e80"
server_key_iv: "7bd27525d2090d642bc2b28940a6f854"
server_key_version: "encryptionkey@0"
I20260925 10:39:36.032716 28146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:39:36.047394 28146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:39:36.051329 28146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:39:36.051594 28146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:39:36.052594 28146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:39:36.053891 28146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:39:36.053998 28146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:36.054082 28146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:39:36.054136 28146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:36.086861 28146 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.1:46335
I20260925 10:39:36.086905 28272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.1:46335 every 8 connection(s)
I20260925 10:39:36.087731 28146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
I20260925 10:39:36.093017 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 28146
I20260925 10:39:36.093240 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/ts-0/wal/instance
I20260925 10:39:36.094019 23952 external_mini_cluster.cc:1428] Setting key 34e790083915a3b0b6549b81dbb274aa
Sep 25 10:39:36 dist-test-slave-dxdd krb5kdc[28041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332776, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.1@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:39:36.108215 28275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:36.090179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:45983 (local address 127.23.100.62:43879)
0925 10:39:36.090650 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:36.090655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:36.091578 (+ 923us) server_negotiation.cc:407] Connection header received
0925 10:39:36.093461 (+ 1883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:36.093469 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:36.093561 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:36.093689 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:36.095378 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:36.096427 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:36.097943 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:36.098671 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:36.098811 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:36.102033 (+ 3222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:36.102041 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:36.102048 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:36.102108 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:36.105088 (+ 2980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:36.105980 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:36.105984 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:36.105986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:36.106113 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:36.106580 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:36.106584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:36.106586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:36.106840 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:36.106948 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:36.107798 (+ 850us) server_negotiation.cc:299] Negotiation successful
0925 10:39:36.107967 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":266,"threads_started":1}
I20260925 10:39:36.109719 28276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:36.090772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:43879 (local address 127.23.100.1:45983)
0925 10:39:36.091425 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:36.091465 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:36.093110 (+ 1645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:36.093862 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:36.093869 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:36.094473 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:36.095219 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:36.095243 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:36.096580 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:36.096583 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:36.097785 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:36.097796 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:36.098863 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:36.098867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:36.099042 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:36.099719 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:36.099744 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:36.101832 (+ 2088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:36.105222 (+ 3390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:36.105233 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:36.105253 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:36.105870 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:36.106224 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:36.106231 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:36.106237 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:36.106478 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:36.106966 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:36.106976 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:36.107429 (+ 453us) client_negotiation.cc:769] Sending connection context
0925 10:39:36.107720 (+ 291us) client_negotiation.cc:240] Negotiation successful
0925 10:39:36.108278 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":259,"threads_started":1}
I20260925 10:39:36.111377 28273 heartbeater.cc:347] Connected to a master server at 127.23.100.62:43879
I20260925 10:39:36.111680 28273 heartbeater.cc:464] Registering TS with master...
I20260925 10:39:36.112324 28273 heartbeater.cc:511] Master 127.23.100.62:43879 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.100.2@KRBTEST.COM; defaulting to no policy
I20260925 10:39:36.114416 28085 ts_manager.cc:194] Registered new tserver with Master: 4f553d0fe89a466cb64c44f36098674f (127.23.100.1:46335)
I20260925 10:39:36.115934 28085 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.1@KRBTEST.COM'} at 127.23.100.1:45983
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab.
Entry for principal kudu/127.23.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab.
I20260925 10:39:36.139009 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.2:0
--local_ip_for_outbound_sockets=127.23.100.2
--webserver_interface=127.23.100.2
--webserver_port=0
--tserver_master_addrs=127.23.100.62:43879
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab
--principal=kudu/127.23.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:39413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:39:36.451336 28279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:39:36.451586 28279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:39:36.451684 28279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:39:36.458582 28279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:39:36.458681 28279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:39:36.458791 28279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.2
I20260925 10:39:36.468519 28279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:39413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab
--principal=kudu/127.23.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.2
--webserver_port=0
--tserver_master_addrs=127.23.100.62:43879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.2
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:39:36.469678 28279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:39:36.470897 28279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:39:36.482925 28287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:36.483006 28285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:36.482981 28284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:36.484022 28279 server_base.cc:1034] running on GCE node
I20260925 10:39:36.484620 28279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:39:36.485482 28279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:39:36.486671 28279 hybrid_clock.cc:648] HybridClock initialized: now 1790332776486626 us; error 36 us; skew 500 ppm
I20260925 10:39:36.486932 28279 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:39:36 dist-test-slave-dxdd krb5kdc[28041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332776, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:39:36.490069 28279 init.cc:377] Logged in from keytab as kudu/127.23.100.2@KRBTEST.COM (short username kudu)
I20260925 10:39:36.491151 28279 fs_manager.cc:359] Metadata directory not provided
I20260925 10:39:36.491235 28279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:39:36.491462 28279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:39:36.494076 28279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/ts-1/data/instance:
uuid: "ad80a33bb47c41cd9ed129663a995b6f"
format_stamp: "Formatted at 2026-09-25 10:39:36 on dist-test-slave-dxdd"
server_key: "8bf27e9afdd999ee1006654e0b8d30a7"
server_key_iv: "f4aff1f4ea3080fccd63089df7b22fb2"
server_key_version: "encryptionkey@0"
I20260925 10:39:36.494889 28279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/ts-1/wal/instance:
uuid: "ad80a33bb47c41cd9ed129663a995b6f"
format_stamp: "Formatted at 2026-09-25 10:39:36 on dist-test-slave-dxdd"
server_key: "8bf27e9afdd999ee1006654e0b8d30a7"
server_key_iv: "f4aff1f4ea3080fccd63089df7b22fb2"
server_key_version: "encryptionkey@0"
I20260925 10:39:36.499580 28279 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260925 10:39:36.502588 28293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:36.503783 28279 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 10:39:36.503903 28279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/ts-1/wal
uuid: "ad80a33bb47c41cd9ed129663a995b6f"
format_stamp: "Formatted at 2026-09-25 10:39:36 on dist-test-slave-dxdd"
server_key: "8bf27e9afdd999ee1006654e0b8d30a7"
server_key_iv: "f4aff1f4ea3080fccd63089df7b22fb2"
server_key_version: "encryptionkey@0"
I20260925 10:39:36.504065 28279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:39:36.517823 28279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:39:36.521319 28279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:39:36.521520 28279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:39:36.522380 28279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:39:36.523542 28279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:39:36.523623 28279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:36.523703 28279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:39:36.523751 28279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:36.557646 28279 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.2:44049
I20260925 10:39:36.557703 28405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.2:44049 every 8 connection(s)
I20260925 10:39:36.558547 28279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
I20260925 10:39:36.566637 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 28279
I20260925 10:39:36.566866 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/ts-1/wal/instance
I20260925 10:39:36.567355 23952 external_mini_cluster.cc:1428] Setting key a1d854b0d7f3b3c43a2c4f6421a71a8d
Sep 25 10:39:36 dist-test-slave-dxdd krb5kdc[28041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332776, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.2@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:39:36.577976 28275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:36.561023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:37791 (local address 127.23.100.62:43879)
0925 10:39:36.561195 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:36.561199 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:36.562257 (+ 1058us) server_negotiation.cc:407] Connection header received
0925 10:39:36.563978 (+ 1721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:36.563983 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:36.564047 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:36.564134 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:36.565491 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:36.566204 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:36.567989 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:36.568532 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:36.568672 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:36.572229 (+ 3557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:36.572234 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:36.572238 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:36.572268 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:36.574969 (+ 2701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:36.575776 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:36.575782 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:36.575785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:36.575916 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:36.576331 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:36.576336 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:36.576339 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:36.576604 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:36.576761 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:36.577414 (+ 653us) server_negotiation.cc:299] Negotiation successful
0925 10:39:36.577639 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 10:39:36.578912 28408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:36.561455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:43879 (local address 127.23.100.2:37791)
0925 10:39:36.562100 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:36.562136 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:36.563733 (+ 1597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:36.564332 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:36.564339 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:36.564834 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:36.565353 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:36.565369 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:36.566374 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:36.566380 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:36.567851 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:36.567863 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:36.568754 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:36.568762 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:36.569022 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:36.569725 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:36.569751 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:36.572012 (+ 2261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:36.575131 (+ 3119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:36.575139 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:36.575156 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:36.575651 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:36.576024 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:36.576030 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:36.576033 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:36.576212 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:36.576729 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:36.576735 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:36.577054 (+ 319us) client_negotiation.cc:769] Sending connection context
0925 10:39:36.577305 (+ 251us) client_negotiation.cc:240] Negotiation successful
0925 10:39:36.577780 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":255,"threads_started":1}
I20260925 10:39:36.580411 28406 heartbeater.cc:347] Connected to a master server at 127.23.100.62:43879
I20260925 10:39:36.580696 28406 heartbeater.cc:464] Registering TS with master...
I20260925 10:39:36.581310 28406 heartbeater.cc:511] Master 127.23.100.62:43879 requested a full tablet report, sending...
I20260925 10:39:36.582834 28085 ts_manager.cc:194] Registered new tserver with Master: ad80a33bb47c41cd9ed129663a995b6f (127.23.100.2:44049)
I20260925 10:39:36.583987 28085 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.2@KRBTEST.COM'} at 127.23.100.2:37791
WARNING: no policy specified for kudu/127.23.100.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab.
Entry for principal kudu/127.23.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab.
I20260925 10:39:36.609091 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.3:0
--local_ip_for_outbound_sockets=127.23.100.3
--webserver_interface=127.23.100.3
--webserver_port=0
--tserver_master_addrs=127.23.100.62:43879
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab
--principal=kudu/127.23.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:39413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:39:36.923448 28411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:39:36.923770 28411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:39:36.923938 28411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:39:36.931257 28411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:39:36.931411 28411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:39:36.931589 28411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.3
I20260925 10:39:36.941406 28411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:39413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab
--principal=kudu/127.23.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.3
--webserver_port=0
--tserver_master_addrs=127.23.100.62:43879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.3
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:39:36.942791 28411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:39:36.944108 28411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:39:36.957139 28416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:36.957109 28417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:36.957258 28419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:36.957883 28411 server_base.cc:1034] running on GCE node
I20260925 10:39:36.958524 28411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:39:36.959399 28411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:39:36.960618 28411 hybrid_clock.cc:648] HybridClock initialized: now 1790332776960538 us; error 49 us; skew 500 ppm
I20260925 10:39:36.960893 28411 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:39:36 dist-test-slave-dxdd krb5kdc[28041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332776, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:39:36.968184 28411 init.cc:377] Logged in from keytab as kudu/127.23.100.3@KRBTEST.COM (short username kudu)
I20260925 10:39:36.969415 28411 fs_manager.cc:359] Metadata directory not provided
I20260925 10:39:36.969506 28411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:39:36.969875 28411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:39:36.972872 28411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/ts-2/data/instance:
uuid: "290f1823121f4e9dbd7d2263aa5fce50"
format_stamp: "Formatted at 2026-09-25 10:39:36 on dist-test-slave-dxdd"
server_key: "ddbcd4bd683cd1a455df9e72621136a2"
server_key_iv: "4cbcbd6eb7158a1741636b1189c1ab0f"
server_key_version: "encryptionkey@0"
I20260925 10:39:36.973659 28411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/ts-2/wal/instance:
uuid: "290f1823121f4e9dbd7d2263aa5fce50"
format_stamp: "Formatted at 2026-09-25 10:39:36 on dist-test-slave-dxdd"
server_key: "ddbcd4bd683cd1a455df9e72621136a2"
server_key_iv: "4cbcbd6eb7158a1741636b1189c1ab0f"
server_key_version: "encryptionkey@0"
I20260925 10:39:36.978863 28411 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260925 10:39:36.982254 28425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:36.983620 28411 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 10:39:36.983793 28411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/ts-2/wal
uuid: "290f1823121f4e9dbd7d2263aa5fce50"
format_stamp: "Formatted at 2026-09-25 10:39:36 on dist-test-slave-dxdd"
server_key: "ddbcd4bd683cd1a455df9e72621136a2"
server_key_iv: "4cbcbd6eb7158a1741636b1189c1ab0f"
server_key_version: "encryptionkey@0"
I20260925 10:39:36.984009 28411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:39:37.019102 28411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:39:37.023347 28411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:39:37.023631 28411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:39:37.024634 28411 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:39:37.026067 28411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:39:37.026188 28411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:37.026350 28411 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:39:37.026443 28411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260925 10:39:37.066985 28411 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.3:37527
I20260925 10:39:37.067080 28537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.3:37527 every 8 connection(s)
I20260925 10:39:37.067962 28411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
I20260925 10:39:37.076108 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 28411
I20260925 10:39:37.076326 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/ts-2/wal/instance
I20260925 10:39:37.076731 23952 external_mini_cluster.cc:1428] Setting key f796fe974216fb8e7ff5b458483b1c88
Sep 25 10:39:37 dist-test-slave-dxdd krb5kdc[28041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332776, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.3@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:39:37.087692 28275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:37.070482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:34325 (local address 127.23.100.62:43879)
0925 10:39:37.070694 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:37.070700 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:37.071942 (+ 1242us) server_negotiation.cc:407] Connection header received
0925 10:39:37.073653 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:37.073658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:37.073728 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:37.073818 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:37.075385 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:37.076256 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:37.077865 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:37.078367 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:37.078508 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:37.082033 (+ 3525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:37.082039 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:37.082050 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:37.082084 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:37.084544 (+ 2460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:37.085247 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:37.085254 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:37.085257 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:37.085383 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:37.085800 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:37.085805 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:37.085808 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:37.086152 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:37.086333 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:37.087050 (+ 717us) server_negotiation.cc:299] Negotiation successful
0925 10:39:37.087281 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 10:39:37.088567 28540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:37.071082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:43879 (local address 127.23.100.3:34325)
0925 10:39:37.071788 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:37.071825 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:37.073410 (+ 1585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:37.074002 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:37.074009 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:37.074674 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:37.075219 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:37.075236 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:37.076415 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:37.076423 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:37.077701 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:37.077713 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:37.078587 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:37.078597 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:37.078876 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:37.079653 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:37.079684 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:37.081820 (+ 2136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:37.084693 (+ 2873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:37.084702 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:37.084716 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:37.085126 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:37.085493 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:37.085499 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:37.085502 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:37.085677 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:37.086375 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:37.086382 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:37.086679 (+ 297us) client_negotiation.cc:769] Sending connection context
0925 10:39:37.086927 (+ 248us) client_negotiation.cc:240] Negotiation successful
0925 10:39:37.087351 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":298,"threads_started":1}
I20260925 10:39:37.089957 28538 heartbeater.cc:347] Connected to a master server at 127.23.100.62:43879
I20260925 10:39:37.090343 28538 heartbeater.cc:464] Registering TS with master...
I20260925 10:39:37.091073 28538 heartbeater.cc:511] Master 127.23.100.62:43879 requested a full tablet report, sending...
I20260925 10:39:37.092680 28085 ts_manager.cc:194] Registered new tserver with Master: 290f1823121f4e9dbd7d2263aa5fce50 (127.23.100.3:37527)
I20260925 10:39:37.093466 28085 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.3@KRBTEST.COM'} at 127.23.100.3:34325
I20260925 10:39:37.104001 23952 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:39:37.119602 28273 heartbeater.cc:503] Master 127.23.100.62:43879 was elected leader, sending a full tablet report...
I20260925 10:39:37.119917 28275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:37.108361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41196 (local address 127.23.100.62:43879)
0925 10:39:37.108524 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:37.108528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:37.109035 (+ 507us) server_negotiation.cc:407] Connection header received
0925 10:39:37.109168 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:37.109172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:37.109232 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:37.109311 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:37.110665 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:37.111342 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:37.112478 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:37.112868 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:37.112958 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:37.115137 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:37.115142 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:37.115145 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:37.115173 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:37.117552 (+ 2379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:37.118146 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:37.118150 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:37.118152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:37.118274 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:37.118637 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:37.118640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:37.118642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:37.118930 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:37.119097 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:37.119408 (+ 311us) server_negotiation.cc:299] Negotiation successful
0925 10:39:37.119603 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 10:39:37.124038 28084 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41196:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:39:37.130785 28084 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:39:37.155104 28551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:37.142748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38066 (local address 127.23.100.2:44049)
0925 10:39:37.144681 (+ 1933us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:37.144694 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:37.144730 (+ 36us) server_negotiation.cc:407] Connection header received
0925 10:39:37.144847 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:37.144858 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:37.145204 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:37.145424 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:39:37.146571 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:37.148656 (+ 2085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:37.152243 (+ 3587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:37.153435 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:37.153997 (+ 562us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:37.154222 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:37.154581 (+ 359us) server_negotiation.cc:299] Negotiation successful
0925 10:39:37.154769 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1736,"thread_start_us":1609,"threads_started":1}
I20260925 10:39:37.155203 28553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:37.142972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.2:44049 (local address 127.0.0.1:38066)
0925 10:39:37.144068 (+ 1096us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:37.144085 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:37.144181 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:37.145758 (+ 1577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:37.145769 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:37.145819 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:39:37.146298 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:37.146336 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:37.150213 (+ 3877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:37.150221 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:37.152073 (+ 1852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:37.152086 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:37.153692 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:37.153700 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:37.153920 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:37.154351 (+ 431us) client_negotiation.cc:769] Sending connection context
0925 10:39:37.154479 (+ 128us) client_negotiation.cc:240] Negotiation successful
0925 10:39:37.154732 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":988,"mutex_wait_us":394,"thread_start_us":823,"threads_started":1}
I20260925 10:39:37.158517 28552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:37.142796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51536 (local address 127.23.100.3:37527)
0925 10:39:37.149357 (+ 6561us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:37.149368 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:37.149405 (+ 37us) server_negotiation.cc:407] Connection header received
0925 10:39:37.149514 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:37.149524 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:37.149792 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:37.149978 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:39:37.151679 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:37.153282 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:37.156038 (+ 2756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:37.157423 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:37.157567 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:37.157796 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:37.157993 (+ 197us) server_negotiation.cc:299] Negotiation successful
0925 10:39:37.158171 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6339,"thread_start_us":6161,"threads_started":1}
I20260925 10:39:37.158540 28550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:37.142210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.3:37527 (local address 127.0.0.1:51536)
0925 10:39:37.143283 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:37.143341 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:37.143527 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:37.149990 (+ 6463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:37.149996 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:37.150028 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:39:37.150650 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:37.150683 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:37.153436 (+ 2753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:37.153442 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:37.155883 (+ 2441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:37.155897 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:37.157629 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:37.157635 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:37.157801 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:37.157838 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 10:39:37.157939 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 10:39:37.158085 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":794,"mutex_wait_us":84,"thread_start_us":640,"threads_started":1}
I20260925 10:39:37.159323 28554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:37.144243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:46335 (local address 127.0.0.1:45954)
0925 10:39:37.150034 (+ 5791us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:37.150051 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:37.150596 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:37.151559 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:37.151564 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:37.151591 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:39:37.152048 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:37.152061 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:37.153937 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:37.153943 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:37.156190 (+ 2247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:37.156203 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:37.158562 (+ 2359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:37.158568 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:37.158728 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:37.158760 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 10:39:37.158852 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 10:39:37.158974 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5635,"mutex_wait_us":53,"thread_start_us":5537,"threads_started":1}
I20260925 10:39:37.159365 28555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:37.144678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45954 (local address 127.23.100.1:46335)
0925 10:39:37.149658 (+ 4980us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:37.149664 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:37.150250 (+ 586us) server_negotiation.cc:407] Connection header received
0925 10:39:37.150908 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:37.150918 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:37.151154 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:37.151320 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:39:37.152280 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:37.153799 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:37.156607 (+ 2808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:37.158033 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:37.158175 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:37.158411 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:37.158900 (+ 489us) server_negotiation.cc:299] Negotiation successful
0925 10:39:37.159074 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4833,"thread_start_us":4678,"threads_started":1}
I20260925 10:39:37.159600 28341 tablet_service.cc:1467] Processing CreateTablet for tablet fd27512e7ba34503bfb9512b349a5916 (DEFAULT_TABLE table=test-table [id=d823e803e31445b5a4263963fef792a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:39:37.160969 28341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd27512e7ba34503bfb9512b349a5916. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:39:37.162252 28473 tablet_service.cc:1467] Processing CreateTablet for tablet fd27512e7ba34503bfb9512b349a5916 (DEFAULT_TABLE table=test-table [id=d823e803e31445b5a4263963fef792a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:39:37.162880 28208 tablet_service.cc:1467] Processing CreateTablet for tablet fd27512e7ba34503bfb9512b349a5916 (DEFAULT_TABLE table=test-table [id=d823e803e31445b5a4263963fef792a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:39:37.163689 28473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd27512e7ba34503bfb9512b349a5916. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:39:37.164176 28208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd27512e7ba34503bfb9512b349a5916. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:39:37.172334 28556 tablet_bootstrap.cc:492] T fd27512e7ba34503bfb9512b349a5916 P ad80a33bb47c41cd9ed129663a995b6f: Bootstrap starting.
I20260925 10:39:37.176002 28556 tablet_bootstrap.cc:654] T fd27512e7ba34503bfb9512b349a5916 P ad80a33bb47c41cd9ed129663a995b6f: Neither blocks nor log segments found. Creating new log.
I20260925 10:39:37.176998 28556 log.cc:824] T fd27512e7ba34503bfb9512b349a5916 P ad80a33bb47c41cd9ed129663a995b6f: Log is configured to *not* fsync() on all Append() calls
I20260925 10:39:37.179780 28556 tablet_bootstrap.cc:492] T fd27512e7ba34503bfb9512b349a5916 P ad80a33bb47c41cd9ed129663a995b6f: No bootstrap required, opened a new log
I20260925 10:39:37.180042 28556 ts_tablet_manager.cc:1397] T fd27512e7ba34503bfb9512b349a5916 P ad80a33bb47c41cd9ed129663a995b6f: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260925 10:39:37.183005 28559 tablet_bootstrap.cc:492] T fd27512e7ba34503bfb9512b349a5916 P 290f1823121f4e9dbd7d2263aa5fce50: Bootstrap starting.
I20260925 10:39:37.183681 28558 tablet_bootstrap.cc:492] T fd27512e7ba34503bfb9512b349a5916 P 4f553d0fe89a466cb64c44f36098674f: Bootstrap starting.
I20260925 10:39:37.186404 28558 tablet_bootstrap.cc:654] T fd27512e7ba34503bfb9512b349a5916 P 4f553d0fe89a466cb64c44f36098674f: Neither blocks nor log segments found. Creating new log.
I20260925 10:39:37.187366 28556 raft_consensus.cc:348] T fd27512e7ba34503bfb9512b349a5916 P ad80a33bb47c41cd9ed129663a995b6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f1823121f4e9dbd7d2263aa5fce50" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37527 } } peers { permanent_uuid: "ad80a33bb47c41cd9ed129663a995b6f" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 44049 } } peers { permanent_uuid: "4f553d0fe89a466cb64c44f36098674f" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 46335 } }
I20260925 10:39:37.187697 28558 log.cc:824] T fd27512e7ba34503bfb9512b349a5916 P 4f553d0fe89a466cb64c44f36098674f: Log is configured to *not* fsync() on all Append() calls
I20260925 10:39:37.187886 28556 raft_consensus.cc:374] T fd27512e7ba34503bfb9512b349a5916 P ad80a33bb47c41cd9ed129663a995b6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:39:37.187997 28556 raft_consensus.cc:729] T fd27512e7ba34503bfb9512b349a5916 P ad80a33bb47c41cd9ed129663a995b6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad80a33bb47c41cd9ed129663a995b6f, State: Initialized, Role: FOLLOWER
I20260925 10:39:37.188733 28556 consensus_queue.cc:260] T fd27512e7ba34503bfb9512b349a5916 P ad80a33bb47c41cd9ed129663a995b6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f1823121f4e9dbd7d2263aa5fce50" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37527 } } peers { permanent_uuid: "ad80a33bb47c41cd9ed129663a995b6f" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 44049 } } peers { permanent_uuid: "4f553d0fe89a466cb64c44f36098674f" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 46335 } }
I20260925 10:39:37.189846 28559 tablet_bootstrap.cc:654] T fd27512e7ba34503bfb9512b349a5916 P 290f1823121f4e9dbd7d2263aa5fce50: Neither blocks nor log segments found. Creating new log.
I20260925 10:39:37.191149 28558 tablet_bootstrap.cc:492] T fd27512e7ba34503bfb9512b349a5916 P 4f553d0fe89a466cb64c44f36098674f: No bootstrap required, opened a new log
I20260925 10:39:37.191219 28559 log.cc:824] T fd27512e7ba34503bfb9512b349a5916 P 290f1823121f4e9dbd7d2263aa5fce50: Log is configured to *not* fsync() on all Append() calls
I20260925 10:39:37.191458 28558 ts_tablet_manager.cc:1397] T fd27512e7ba34503bfb9512b349a5916 P 4f553d0fe89a466cb64c44f36098674f: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20260925 10:39:37.192613 28406 heartbeater.cc:503] Master 127.23.100.62:43879 was elected leader, sending a full tablet report...
I20260925 10:39:37.193217 28556 ts_tablet_manager.cc:1428] T fd27512e7ba34503bfb9512b349a5916 P ad80a33bb47c41cd9ed129663a995b6f: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260925 10:39:37.195094 28559 tablet_bootstrap.cc:492] T fd27512e7ba34503bfb9512b349a5916 P 290f1823121f4e9dbd7d2263aa5fce50: No bootstrap required, opened a new log
I20260925 10:39:37.195369 28559 ts_tablet_manager.cc:1397] T fd27512e7ba34503bfb9512b349a5916 P 290f1823121f4e9dbd7d2263aa5fce50: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20260925 10:39:37.200258 28558 raft_consensus.cc:348] T fd27512e7ba34503bfb9512b349a5916 P 4f553d0fe89a466cb64c44f36098674f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f1823121f4e9dbd7d2263aa5fce50" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37527 } } peers { permanent_uuid: "ad80a33bb47c41cd9ed129663a995b6f" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 44049 } } peers { permanent_uuid: "4f553d0fe89a466cb64c44f36098674f" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 46335 } }
I20260925 10:39:37.200598 28558 raft_consensus.cc:374] T fd27512e7ba34503bfb9512b349a5916 P 4f553d0fe89a466cb64c44f36098674f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:39:37.200696 28558 raft_consensus.cc:729] T fd27512e7ba34503bfb9512b349a5916 P 4f553d0fe89a466cb64c44f36098674f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f553d0fe89a466cb64c44f36098674f, State: Initialized, Role: FOLLOWER
I20260925 10:39:37.201361 28558 consensus_queue.cc:260] T fd27512e7ba34503bfb9512b349a5916 P 4f553d0fe89a466cb64c44f36098674f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f1823121f4e9dbd7d2263aa5fce50" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37527 } } peers { permanent_uuid: "ad80a33bb47c41cd9ed129663a995b6f" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 44049 } } peers { permanent_uuid: "4f553d0fe89a466cb64c44f36098674f" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 46335 } }
I20260925 10:39:37.202934 28558 ts_tablet_manager.cc:1428] T fd27512e7ba34503bfb9512b349a5916 P 4f553d0fe89a466cb64c44f36098674f: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260925 10:39:37.203662 28559 raft_consensus.cc:348] T fd27512e7ba34503bfb9512b349a5916 P 290f1823121f4e9dbd7d2263aa5fce50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f1823121f4e9dbd7d2263aa5fce50" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37527 } } peers { permanent_uuid: "ad80a33bb47c41cd9ed129663a995b6f" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 44049 } } peers { permanent_uuid: "4f553d0fe89a466cb64c44f36098674f" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 46335 } }
I20260925 10:39:37.204043 28559 raft_consensus.cc:374] T fd27512e7ba34503bfb9512b349a5916 P 290f1823121f4e9dbd7d2263aa5fce50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:39:37.204164 28559 raft_consensus.cc:729] T fd27512e7ba34503bfb9512b349a5916 P 290f1823121f4e9dbd7d2263aa5fce50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 290f1823121f4e9dbd7d2263aa5fce50, State: Initialized, Role: FOLLOWER
I20260925 10:39:37.204732 28559 consensus_queue.cc:260] T fd27512e7ba34503bfb9512b349a5916 P 290f1823121f4e9dbd7d2263aa5fce50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f1823121f4e9dbd7d2263aa5fce50" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37527 } } peers { permanent_uuid: "ad80a33bb47c41cd9ed129663a995b6f" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 44049 } } peers { permanent_uuid: "4f553d0fe89a466cb64c44f36098674f" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 46335 } }
I20260925 10:39:37.205940 28538 heartbeater.cc:503] Master 127.23.100.62:43879 was elected leader, sending a full tablet report...
I20260925 10:39:37.206454 28559 ts_tablet_manager.cc:1428] T fd27512e7ba34503bfb9512b349a5916 P 290f1823121f4e9dbd7d2263aa5fce50: Time spent starting tablet: real 0.011s user 0.007s sys 0.005s
I20260925 10:39:37.301718 28561 raft_consensus.cc:482] T fd27512e7ba34503bfb9512b349a5916 P ad80a33bb47c41cd9ed129663a995b6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:39:37.302062 28561 raft_consensus.cc:504] T fd27512e7ba34503bfb9512b349a5916 P ad80a33bb47c41cd9ed129663a995b6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f1823121f4e9dbd7d2263aa5fce50" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37527 } } peers { permanent_uuid: "ad80a33bb47c41cd9ed129663a995b6f" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 44049 } } peers { permanent_uuid: "4f553d0fe89a466cb64c44f36098674f" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 46335 } }
I20260925 10:39:37.303949 28561 leader_election.cc:288] T fd27512e7ba34503bfb9512b349a5916 P ad80a33bb47c41cd9ed129663a995b6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 290f1823121f4e9dbd7d2263aa5fce50 (127.23.100.3:37527), 4f553d0fe89a466cb64c44f36098674f (127.23.100.1:46335)
W20260925 10:39:37.310787 28407 tablet.cc:2369] T fd27512e7ba34503bfb9512b349a5916 P ad80a33bb47c41cd9ed129663a995b6f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:39:37.311223 28552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:37.304751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:37373 (local address 127.23.100.3:37527)
0925 10:39:37.304963 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:37.304971 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:37.305529 (+ 558us) server_negotiation.cc:407] Connection header received
0925 10:39:37.305801 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:37.305805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:37.305880 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:37.305978 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:39:37.306648 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:37.307477 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:37.309422 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:37.310202 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:37.310414 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:37.310490 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:37.310837 (+ 347us) server_negotiation.cc:299] Negotiation successful
0925 10:39:37.310956 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 10:39:37.311321 28555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:37.304463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:54673 (local address 127.23.100.1:46335)
0925 10:39:37.304690 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:37.304699 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:37.305569 (+ 870us) server_negotiation.cc:407] Connection header received
0925 10:39:37.305856 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:37.305865 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:37.305923 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:37.306037 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:39:37.306836 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:37.307586 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:37.309410 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:37.310164 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:37.310396 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:37.310475 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:37.310939 (+ 464us) server_negotiation.cc:299] Negotiation successful
0925 10:39:37.311066 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 10:39:37.311338 28566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:37.304265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.3:37527 (local address 127.23.100.2:37373)
0925 10:39:37.305412 (+ 1147us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:37.305427 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:37.305610 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:37.305995 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:37.306000 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:37.306023 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:39:37.306483 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:37.306492 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:37.307604 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:37.307611 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:37.309271 (+ 1660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:37.309283 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:37.310468 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:37.310474 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:37.310661 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:37.310703 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 10:39:37.310813 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 10:39:37.310943 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":991,"mutex_wait_us":35,"thread_start_us":881,"threads_started":1}
I20260925 10:39:37.312028 28565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:37.304347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:46335 (local address 127.23.100.2:54673)
0925 10:39:37.305453 (+ 1106us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:37.305467 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:37.305679 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:37.306037 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:37.306042 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:37.306063 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:39:37.306665 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:37.306678 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:37.307736 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:37.307741 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:37.309260 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:37.309271 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:37.310441 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:37.310451 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:37.310719 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:37.310768 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 10:39:37.311483 (+ 715us) client_negotiation.cc:240] Negotiation successful
0925 10:39:37.311613 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1013,"mutex_wait_us":122,"spinlock_wait_cycles":24960,"thread_start_us":551,"threads_started":1}
I20260925 10:39:37.312636 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd27512e7ba34503bfb9512b349a5916" candidate_uuid: "ad80a33bb47c41cd9ed129663a995b6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "290f1823121f4e9dbd7d2263aa5fce50" is_pre_election: true
I20260925 10:39:37.312922 28228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd27512e7ba34503bfb9512b349a5916" candidate_uuid: "ad80a33bb47c41cd9ed129663a995b6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f553d0fe89a466cb64c44f36098674f" is_pre_election: true
I20260925 10:39:37.313109 28493 raft_consensus.cc:2454] T fd27512e7ba34503bfb9512b349a5916 P 290f1823121f4e9dbd7d2263aa5fce50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad80a33bb47c41cd9ed129663a995b6f in term 0.
I20260925 10:39:37.313423 28228 raft_consensus.cc:2454] T fd27512e7ba34503bfb9512b349a5916 P 4f553d0fe89a466cb64c44f36098674f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad80a33bb47c41cd9ed129663a995b6f in term 0.
I20260925 10:39:37.313778 28295 leader_election.cc:302] T fd27512e7ba34503bfb9512b349a5916 P ad80a33bb47c41cd9ed129663a995b6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 290f1823121f4e9dbd7d2263aa5fce50, ad80a33bb47c41cd9ed129663a995b6f; no voters:
I20260925 10:39:37.314167 28561 raft_consensus.cc:2781] T fd27512e7ba34503bfb9512b349a5916 P ad80a33bb47c41cd9ed129663a995b6f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:39:37.314352 28561 raft_consensus.cc:482] T fd27512e7ba34503bfb9512b349a5916 P ad80a33bb47c41cd9ed129663a995b6f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:39:37.314461 28561 raft_consensus.cc:3037] T fd27512e7ba34503bfb9512b349a5916 P ad80a33bb47c41cd9ed129663a995b6f [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:39:37.316664 28561 raft_consensus.cc:504] T fd27512e7ba34503bfb9512b349a5916 P ad80a33bb47c41cd9ed129663a995b6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f1823121f4e9dbd7d2263aa5fce50" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37527 } } peers { permanent_uuid: "ad80a33bb47c41cd9ed129663a995b6f" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 44049 } } peers { permanent_uuid: "4f553d0fe89a466cb64c44f36098674f" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 46335 } }
I20260925 10:39:37.317243 28561 leader_election.cc:288] T fd27512e7ba34503bfb9512b349a5916 P ad80a33bb47c41cd9ed129663a995b6f [CANDIDATE]: Term 1 election: Requested vote from peers 290f1823121f4e9dbd7d2263aa5fce50 (127.23.100.3:37527), 4f553d0fe89a466cb64c44f36098674f (127.23.100.1:46335)
I20260925 10:39:37.317771 28228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd27512e7ba34503bfb9512b349a5916" candidate_uuid: "ad80a33bb47c41cd9ed129663a995b6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f553d0fe89a466cb64c44f36098674f"
I20260925 10:39:37.317798 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd27512e7ba34503bfb9512b349a5916" candidate_uuid: "ad80a33bb47c41cd9ed129663a995b6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "290f1823121f4e9dbd7d2263aa5fce50"
I20260925 10:39:37.317992 28493 raft_consensus.cc:3037] T fd27512e7ba34503bfb9512b349a5916 P 290f1823121f4e9dbd7d2263aa5fce50 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:39:37.317988 28228 raft_consensus.cc:3037] T fd27512e7ba34503bfb9512b349a5916 P 4f553d0fe89a466cb64c44f36098674f [term 0 FOLLOWER]: Advancing to term 1
W20260925 10:39:37.319434 28539 tablet.cc:2369] T fd27512e7ba34503bfb9512b349a5916 P 290f1823121f4e9dbd7d2263aa5fce50: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:39:37.321272 28228 raft_consensus.cc:2454] T fd27512e7ba34503bfb9512b349a5916 P 4f553d0fe89a466cb64c44f36098674f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad80a33bb47c41cd9ed129663a995b6f in term 1.
I20260925 10:39:37.321272 28493 raft_consensus.cc:2454] T fd27512e7ba34503bfb9512b349a5916 P 290f1823121f4e9dbd7d2263aa5fce50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad80a33bb47c41cd9ed129663a995b6f in term 1.
I20260925 10:39:37.321719 28296 leader_election.cc:302] T fd27512e7ba34503bfb9512b349a5916 P ad80a33bb47c41cd9ed129663a995b6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 290f1823121f4e9dbd7d2263aa5fce50, 4f553d0fe89a466cb64c44f36098674f, ad80a33bb47c41cd9ed129663a995b6f; no voters:
I20260925 10:39:37.322058 28561 raft_consensus.cc:2781] T fd27512e7ba34503bfb9512b349a5916 P ad80a33bb47c41cd9ed129663a995b6f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:39:37.322535 28561 raft_consensus.cc:686] T fd27512e7ba34503bfb9512b349a5916 P ad80a33bb47c41cd9ed129663a995b6f [term 1 LEADER]: Becoming Leader. State: Replica: ad80a33bb47c41cd9ed129663a995b6f, State: Running, Role: LEADER
I20260925 10:39:37.322986 28561 consensus_queue.cc:237] T fd27512e7ba34503bfb9512b349a5916 P ad80a33bb47c41cd9ed129663a995b6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f1823121f4e9dbd7d2263aa5fce50" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37527 } } peers { permanent_uuid: "ad80a33bb47c41cd9ed129663a995b6f" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 44049 } } peers { permanent_uuid: "4f553d0fe89a466cb64c44f36098674f" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 46335 } }
I20260925 10:39:37.327466 28085 catalog_manager.cc:5495] T fd27512e7ba34503bfb9512b349a5916 P ad80a33bb47c41cd9ed129663a995b6f reported cstate change: term changed from 0 to 1, leader changed from <none> to ad80a33bb47c41cd9ed129663a995b6f (127.23.100.2). New cstate: current_term: 1 leader_uuid: "ad80a33bb47c41cd9ed129663a995b6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f1823121f4e9dbd7d2263aa5fce50" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad80a33bb47c41cd9ed129663a995b6f" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f553d0fe89a466cb64c44f36098674f" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 46335 } health_report { overall_health: UNKNOWN } } }
W20260925 10:39:37.339841 28274 tablet.cc:2369] T fd27512e7ba34503bfb9512b349a5916 P 4f553d0fe89a466cb64c44f36098674f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:39:37.352566 28551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:37.347540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38068 (local address 127.23.100.2:44049)
0925 10:39:37.347727 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:37.347732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:37.347886 (+ 154us) server_negotiation.cc:407] Connection header received
0925 10:39:37.348047 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:37.348050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:37.348126 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:37.348258 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:39:37.348556 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:37.349250 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:37.350595 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:37.350980 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:37.351070 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:37.351481 (+ 411us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:39:37.352092 (+ 611us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:39:37.352214 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:37.352311 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 10:39:37.352372 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 10:39:37.360610 28228 raft_consensus.cc:1261] T fd27512e7ba34503bfb9512b349a5916 P 4f553d0fe89a466cb64c44f36098674f [term 1 FOLLOWER]: Refusing update from remote peer ad80a33bb47c41cd9ed129663a995b6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:39:37.360610 28493 raft_consensus.cc:1261] T fd27512e7ba34503bfb9512b349a5916 P 290f1823121f4e9dbd7d2263aa5fce50 [term 1 FOLLOWER]: Refusing update from remote peer ad80a33bb47c41cd9ed129663a995b6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:39:37.361635 28567 consensus_queue.cc:1035] T fd27512e7ba34503bfb9512b349a5916 P ad80a33bb47c41cd9ed129663a995b6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f553d0fe89a466cb64c44f36098674f" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 46335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:39:37.361979 28561 consensus_queue.cc:1035] T fd27512e7ba34503bfb9512b349a5916 P ad80a33bb47c41cd9ed129663a995b6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "290f1823121f4e9dbd7d2263aa5fce50" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:39:37.372941 28571 mvcc.cc:205] Tried to move back new op lower bound from 7333203056057638912 to 7333203055918206976. Current Snapshot: MvccSnapshot[applied={T|T < 7333203056057638912}]
I20260925 10:39:37.375799 28572 mvcc.cc:205] Tried to move back new op lower bound from 7333203056057638912 to 7333203055918206976. Current Snapshot: MvccSnapshot[applied={T|T < 7333203056057638912}]
I20260925 10:39:37.385624 28084 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:39:37.385936 28084 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:39:37.389504 28084 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 91202005ee564d6aa380909adebda070: Sending DeleteTablet for 3 replicas of tablet fd27512e7ba34503bfb9512b349a5916
I20260925 10:39:37.390623 28473 tablet_service.cc:1514] Processing DeleteTablet for tablet fd27512e7ba34503bfb9512b349a5916 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:39:37 UTC) from {username='kudu', principal='kudu/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:51536
I20260925 10:39:37.390748 28341 tablet_service.cc:1514] Processing DeleteTablet for tablet fd27512e7ba34503bfb9512b349a5916 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:39:37 UTC) from {username='kudu', principal='kudu/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:38066
I20260925 10:39:37.391405 28208 tablet_service.cc:1514] Processing DeleteTablet for tablet fd27512e7ba34503bfb9512b349a5916 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:39:37 UTC) from {username='kudu', principal='kudu/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:45954
I20260925 10:39:37.391782 28580 tablet_replica.cc:330] T fd27512e7ba34503bfb9512b349a5916 P ad80a33bb47c41cd9ed129663a995b6f: stopping tablet replica
I20260925 10:39:37.392318 28580 raft_consensus.cc:2229] T fd27512e7ba34503bfb9512b349a5916 P ad80a33bb47c41cd9ed129663a995b6f [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:39:37.393149 28580 raft_consensus.cc:2258] T fd27512e7ba34503bfb9512b349a5916 P ad80a33bb47c41cd9ed129663a995b6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:39:37.395639 28580 ts_tablet_manager.cc:1905] T fd27512e7ba34503bfb9512b349a5916 P ad80a33bb47c41cd9ed129663a995b6f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:39:37.396903 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 28146
I20260925 10:39:37.405591 28580 ts_tablet_manager.cc:1918] T fd27512e7ba34503bfb9512b349a5916 P ad80a33bb47c41cd9ed129663a995b6f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:39:37.405889 28580 log.cc:1196] T fd27512e7ba34503bfb9512b349a5916 P ad80a33bb47c41cd9ed129663a995b6f: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/ts-1/wal/wals/fd27512e7ba34503bfb9512b349a5916
I20260925 10:39:37.407318 28580 ts_tablet_manager.cc:1939] T fd27512e7ba34503bfb9512b349a5916 P ad80a33bb47c41cd9ed129663a995b6f: Deleting consensus metadata
I20260925 10:39:37.409637 28071 catalog_manager.cc:4841] TS ad80a33bb47c41cd9ed129663a995b6f (127.23.100.2:44049): tablet fd27512e7ba34503bfb9512b349a5916 (table test-table [id=d823e803e31445b5a4263963fef792a8]) successfully deleted
I20260925 10:39:37.411708 28581 tablet_replica.cc:330] T fd27512e7ba34503bfb9512b349a5916 P 4f553d0fe89a466cb64c44f36098674f: stopping tablet replica
I20260925 10:39:37.412320 28581 raft_consensus.cc:2229] T fd27512e7ba34503bfb9512b349a5916 P 4f553d0fe89a466cb64c44f36098674f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:39:37.413000 28581 raft_consensus.cc:2258] T fd27512e7ba34503bfb9512b349a5916 P 4f553d0fe89a466cb64c44f36098674f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:39:37.414544 28579 tablet_replica.cc:330] T fd27512e7ba34503bfb9512b349a5916 P 290f1823121f4e9dbd7d2263aa5fce50: stopping tablet replica
I20260925 10:39:37.414944 28579 raft_consensus.cc:2229] T fd27512e7ba34503bfb9512b349a5916 P 290f1823121f4e9dbd7d2263aa5fce50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:39:37.415410 28579 raft_consensus.cc:2258] T fd27512e7ba34503bfb9512b349a5916 P 290f1823121f4e9dbd7d2263aa5fce50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:39:37.415657 28581 ts_tablet_manager.cc:1905] T fd27512e7ba34503bfb9512b349a5916 P 4f553d0fe89a466cb64c44f36098674f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:39:37.417196 28579 ts_tablet_manager.cc:1905] T fd27512e7ba34503bfb9512b349a5916 P 290f1823121f4e9dbd7d2263aa5fce50: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 25 10:39:37 dist-test-slave-dxdd krb5kdc[28041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.1@KRBTEST.COM
I20260925 10:39:37.423197 28581 ts_tablet_manager.cc:1918] T fd27512e7ba34503bfb9512b349a5916 P 4f553d0fe89a466cb64c44f36098674f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:39:37.423408 28581 log.cc:1196] T fd27512e7ba34503bfb9512b349a5916 P 4f553d0fe89a466cb64c44f36098674f: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/ts-0/wal/wals/fd27512e7ba34503bfb9512b349a5916
I20260925 10:39:37.423889 28579 ts_tablet_manager.cc:1918] T fd27512e7ba34503bfb9512b349a5916 P 290f1823121f4e9dbd7d2263aa5fce50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:39:37.424028 28581 ts_tablet_manager.cc:1939] T fd27512e7ba34503bfb9512b349a5916 P 4f553d0fe89a466cb64c44f36098674f: Deleting consensus metadata
I20260925 10:39:37.424050 28579 log.cc:1196] T fd27512e7ba34503bfb9512b349a5916 P 290f1823121f4e9dbd7d2263aa5fce50: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790332740062128-23952-0/minicluster-data/ts-2/wal/wals/fd27512e7ba34503bfb9512b349a5916
I20260925 10:39:37.424553 28579 ts_tablet_manager.cc:1939] T fd27512e7ba34503bfb9512b349a5916 P 290f1823121f4e9dbd7d2263aa5fce50: Deleting consensus metadata
I20260925 10:39:37.426083 28070 catalog_manager.cc:4841] TS 290f1823121f4e9dbd7d2263aa5fce50 (127.23.100.3:37527): tablet fd27512e7ba34503bfb9512b349a5916 (table test-table [id=d823e803e31445b5a4263963fef792a8]) successfully deleted
I20260925 10:39:37.426139 28071 catalog_manager.cc:4841] TS 4f553d0fe89a466cb64c44f36098674f (127.23.100.1:46335): tablet fd27512e7ba34503bfb9512b349a5916 (table test-table [id=d823e803e31445b5a4263963fef792a8]) successfully deleted
I20260925 10:39:37.430719 28555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:37.398251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45968 (local address 127.23.100.1:46335)
0925 10:39:37.398440 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:37.398447 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:37.410924 (+ 12477us) server_negotiation.cc:407] Connection header received
0925 10:39:37.411154 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:37.411160 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:37.411259 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:37.411376 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:37.413117 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:37.414339 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:37.416678 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:37.417355 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:37.417458 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:37.423996 (+ 6538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:37.424011 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:37.424028 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:37.424076 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:37.427955 (+ 3879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:37.428773 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:37.428779 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:37.428781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:37.428905 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:37.429316 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:37.429319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:37.429320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:37.429626 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:37.429848 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:37.430209 (+ 361us) server_negotiation.cc:299] Negotiation successful
0925 10:39:37.430428 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 10:39:37.431345 28268 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:39:37.563432 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 28146
I20260925 10:39:37.584296 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 28279
Sep 25 10:39:37 dist-test-slave-dxdd krb5kdc[28041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.2@KRBTEST.COM
I20260925 10:39:37.601648 28551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:37.585090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38070 (local address 127.23.100.2:44049)
0925 10:39:37.585277 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:37.585285 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:37.585405 (+ 120us) server_negotiation.cc:407] Connection header received
0925 10:39:37.585578 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:37.585584 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:37.585685 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:37.585799 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:37.587325 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:37.588699 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:37.590655 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:37.591102 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:37.591198 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:37.595884 (+ 4686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:37.595895 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:37.595909 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:37.595943 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:37.598935 (+ 2992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:37.599579 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:37.599587 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:37.599589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:37.599728 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:37.600126 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:37.600130 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:37.600132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:37.600465 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:37.600779 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:37.601057 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 10:39:37.601305 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 10:39:37.602558 28401 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:39:37.714875 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 28279
I20260925 10:39:37.733394 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 28411
Sep 25 10:39:37 dist-test-slave-dxdd krb5kdc[28041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.3@KRBTEST.COM
I20260925 10:39:37.747131 28552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:37.734176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51538 (local address 127.23.100.3:37527)
0925 10:39:37.734421 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:37.734426 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:37.734460 (+ 34us) server_negotiation.cc:407] Connection header received
0925 10:39:37.734607 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:37.734611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:37.734671 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:37.734756 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:37.735893 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:37.736603 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:37.737776 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:37.738166 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:37.738255 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:37.742243 (+ 3988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:37.742256 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:37.742271 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:37.742345 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:37.744790 (+ 2445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:37.745333 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:37.745339 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:37.745341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:37.745480 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:37.745783 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:37.745786 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:37.745788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:37.746090 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:37.746374 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:37.746630 (+ 256us) server_negotiation.cc:299] Negotiation successful
0925 10:39:37.746789 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 10:39:37.747747 28533 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:39:37.835987 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 28411
I20260925 10:39:37.852249 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 28054
I20260925 10:39:37.853354 28115 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:39:37.962271 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 28054
2026-09-25T10:39:37Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4135 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T10:39:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:39:38Z Disabled control of system clock
I20260925 10:39:38.028143 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:35339
--webserver_interface=127.23.100.62
--webserver_port=0
--builtin_ntp_servers=127.23.100.20:35399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.62:35339
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 10:39:38.340082 28590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:39:38.340368 28590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:39:38.340453 28590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:39:38.347477 28590 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:39:38.347573 28590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:39:38.347610 28590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:39:38.347669 28590 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:39:38.347714 28590 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:39:38.359624 28590 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:35399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.62:35339
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:35339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:39:38.360832 28590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:39:38.362573 28590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:39:38.374141 28595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:38.374155 28596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:38.374492 28590 server_base.cc:1034] running on GCE node
W20260925 10:39:38.374181 28598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:38.375484 28590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:39:38.376803 28590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:39:38.378036 28590 hybrid_clock.cc:648] HybridClock initialized: now 1790332778378003 us; error 45 us; skew 500 ppm
I20260925 10:39:38.378455 28590 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:39:38.380633 28590 webserver.cc:460] Webserver started at http://127.23.100.62:32803/ using document root <none> and password file <none>
I20260925 10:39:38.381325 28590 fs_manager.cc:359] Metadata directory not provided
I20260925 10:39:38.381456 28590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:39:38.381780 28590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:39:38.384692 28590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/master-0/data/instance:
uuid: "d063adba002c4890ab69c4aff9b886ae"
format_stamp: "Formatted at 2026-09-25 10:39:38 on dist-test-slave-dxdd"
server_key: "f44b53a47bb6787be5ad0269f0b04f3c"
server_key_iv: "902df3dd5f24c8b5635aea317d629745"
server_key_version: "encryptionkey@0"
I20260925 10:39:38.385481 28590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/master-0/wal/instance:
uuid: "d063adba002c4890ab69c4aff9b886ae"
format_stamp: "Formatted at 2026-09-25 10:39:38 on dist-test-slave-dxdd"
server_key: "f44b53a47bb6787be5ad0269f0b04f3c"
server_key_iv: "902df3dd5f24c8b5635aea317d629745"
server_key_version: "encryptionkey@0"
I20260925 10:39:38.390489 28590 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 10:39:38.393661 28604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:38.394961 28590 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:39:38.395143 28590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/master-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/master-0/wal
uuid: "d063adba002c4890ab69c4aff9b886ae"
format_stamp: "Formatted at 2026-09-25 10:39:38 on dist-test-slave-dxdd"
server_key: "f44b53a47bb6787be5ad0269f0b04f3c"
server_key_iv: "902df3dd5f24c8b5635aea317d629745"
server_key_version: "encryptionkey@0"
I20260925 10:39:38.395364 28590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:39:38.400926 28590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:39:38.401659 28590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:39:38.401953 28590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:39:38.419703 28590 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.62:35339
I20260925 10:39:38.419745 28655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.62:35339 every 8 connection(s)
I20260925 10:39:38.421106 28590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
I20260925 10:39:38.424562 28656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:39:38.426623 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 28590
I20260925 10:39:38.426822 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/master-0/wal/instance
I20260925 10:39:38.427438 23952 external_mini_cluster.cc:1428] Setting key de61798e519c5251cf872843da9a6516
I20260925 10:39:38.435096 28659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:38.429132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33148 (local address 127.23.100.62:35339)
0925 10:39:38.430077 (+ 945us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:38.430101 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:38.430140 (+ 39us) server_negotiation.cc:407] Connection header received
0925 10:39:38.431757 (+ 1617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:38.431776 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:38.432152 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:38.432456 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:38.433537 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:38.433543 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:38.433555 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:39:38.433571 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:38.433632 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:39:38.433649 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:38.433853 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:38.433951 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 10:39:38.434204 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":374,"threads_started":1}
I20260925 10:39:38.436467 28656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d063adba002c4890ab69c4aff9b886ae: Bootstrap starting.
I20260925 10:39:38.439512 28656 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d063adba002c4890ab69c4aff9b886ae: Neither blocks nor log segments found. Creating new log.
I20260925 10:39:38.440481 28656 log.cc:824] T 00000000000000000000000000000000 P d063adba002c4890ab69c4aff9b886ae: Log is configured to *not* fsync() on all Append() calls
I20260925 10:39:38.442991 28656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d063adba002c4890ab69c4aff9b886ae: No bootstrap required, opened a new log
I20260925 10:39:38.449982 28656 raft_consensus.cc:348] T 00000000000000000000000000000000 P d063adba002c4890ab69c4aff9b886ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d063adba002c4890ab69c4aff9b886ae" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 35339 } }
I20260925 10:39:38.450358 28656 raft_consensus.cc:374] T 00000000000000000000000000000000 P d063adba002c4890ab69c4aff9b886ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:39:38.450469 28656 raft_consensus.cc:729] T 00000000000000000000000000000000 P d063adba002c4890ab69c4aff9b886ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d063adba002c4890ab69c4aff9b886ae, State: Initialized, Role: FOLLOWER
I20260925 10:39:38.451119 28656 consensus_queue.cc:260] T 00000000000000000000000000000000 P d063adba002c4890ab69c4aff9b886ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d063adba002c4890ab69c4aff9b886ae" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 35339 } }
I20260925 10:39:38.451323 28656 raft_consensus.cc:388] T 00000000000000000000000000000000 P d063adba002c4890ab69c4aff9b886ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:39:38.451416 28656 raft_consensus.cc:482] T 00000000000000000000000000000000 P d063adba002c4890ab69c4aff9b886ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:39:38.451529 28656 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d063adba002c4890ab69c4aff9b886ae [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:39:38.453466 28656 raft_consensus.cc:504] T 00000000000000000000000000000000 P d063adba002c4890ab69c4aff9b886ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d063adba002c4890ab69c4aff9b886ae" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 35339 } }
I20260925 10:39:38.453920 28656 leader_election.cc:302] T 00000000000000000000000000000000 P d063adba002c4890ab69c4aff9b886ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d063adba002c4890ab69c4aff9b886ae; no voters:
I20260925 10:39:38.454555 28656 leader_election.cc:288] T 00000000000000000000000000000000 P d063adba002c4890ab69c4aff9b886ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:39:38.454691 28661 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d063adba002c4890ab69c4aff9b886ae [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:39:38.455399 28661 raft_consensus.cc:686] T 00000000000000000000000000000000 P d063adba002c4890ab69c4aff9b886ae [term 1 LEADER]: Becoming Leader. State: Replica: d063adba002c4890ab69c4aff9b886ae, State: Running, Role: LEADER
I20260925 10:39:38.455899 28661 consensus_queue.cc:237] T 00000000000000000000000000000000 P d063adba002c4890ab69c4aff9b886ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d063adba002c4890ab69c4aff9b886ae" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 35339 } }
I20260925 10:39:38.456501 28656 sys_catalog.cc:564] T 00000000000000000000000000000000 P d063adba002c4890ab69c4aff9b886ae [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:39:38.458766 28662 sys_catalog.cc:455] T 00000000000000000000000000000000 P d063adba002c4890ab69c4aff9b886ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d063adba002c4890ab69c4aff9b886ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d063adba002c4890ab69c4aff9b886ae" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 35339 } } }
I20260925 10:39:38.459121 28662 sys_catalog.cc:458] T 00000000000000000000000000000000 P d063adba002c4890ab69c4aff9b886ae [sys.catalog]: This master's current role is: LEADER
I20260925 10:39:38.458981 28663 sys_catalog.cc:455] T 00000000000000000000000000000000 P d063adba002c4890ab69c4aff9b886ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader d063adba002c4890ab69c4aff9b886ae. Latest consensus state: current_term: 1 leader_uuid: "d063adba002c4890ab69c4aff9b886ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d063adba002c4890ab69c4aff9b886ae" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 35339 } } }
I20260925 10:39:38.459396 28663 sys_catalog.cc:458] T 00000000000000000000000000000000 P d063adba002c4890ab69c4aff9b886ae [sys.catalog]: This master's current role is: LEADER
I20260925 10:39:38.461488 28668 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:39:38.465950 28668 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:39:38.473654 28668 catalog_manager.cc:1347] Generated new cluster ID: e6a2fdb3cea547eeb48bc94394baf620
I20260925 10:39:38.473763 28668 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:39:38.525847 28668 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:39:38.526271 28668 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:39:38.538293 28668 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d063adba002c4890ab69c4aff9b886ae: Generated new TSK 0
I20260925 10:39:38.539176 28668 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:39:38.551054 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.1:0
--local_ip_for_outbound_sockets=127.23.100.1
--webserver_interface=127.23.100.1
--webserver_port=0
--tserver_master_addrs=127.23.100.62:35339
--builtin_ntp_servers=127.23.100.20:35399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 10:39:38.871459 28680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:39:38.871706 28680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:39:38.871852 28680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:39:38.878953 28680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:39:38.879040 28680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:39:38.879217 28680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.1
I20260925 10:39:38.887996 28680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:35399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.1
--webserver_port=0
--tserver_master_addrs=127.23.100.62:35339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.1
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:39:38.889187 28680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:39:38.890843 28680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:39:38.903699 28685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:38.903779 28688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:38.903770 28686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:38.904822 28680 server_base.cc:1034] running on GCE node
I20260925 10:39:38.905490 28680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:39:38.906405 28680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:39:38.907625 28680 hybrid_clock.cc:648] HybridClock initialized: now 1790332778907569 us; error 47 us; skew 500 ppm
I20260925 10:39:38.907903 28680 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:39:38.910132 28680 webserver.cc:460] Webserver started at http://127.23.100.1:38585/ using document root <none> and password file <none>
I20260925 10:39:38.910897 28680 fs_manager.cc:359] Metadata directory not provided
I20260925 10:39:38.910983 28680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:39:38.911257 28680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:39:38.914108 28680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/ts-0/data/instance:
uuid: "f781e97a08174249a37b682c48d36189"
format_stamp: "Formatted at 2026-09-25 10:39:38 on dist-test-slave-dxdd"
server_key: "54704d64f17ddd2721d8fea27d4967e7"
server_key_iv: "4086d59dc8ed26036a0752572448080b"
server_key_version: "encryptionkey@0"
I20260925 10:39:38.914884 28680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/ts-0/wal/instance:
uuid: "f781e97a08174249a37b682c48d36189"
format_stamp: "Formatted at 2026-09-25 10:39:38 on dist-test-slave-dxdd"
server_key: "54704d64f17ddd2721d8fea27d4967e7"
server_key_iv: "4086d59dc8ed26036a0752572448080b"
server_key_version: "encryptionkey@0"
I20260925 10:39:38.919560 28680 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 10:39:38.922880 28694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:38.924052 28680 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260925 10:39:38.924177 28680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/ts-0/wal
uuid: "f781e97a08174249a37b682c48d36189"
format_stamp: "Formatted at 2026-09-25 10:39:38 on dist-test-slave-dxdd"
server_key: "54704d64f17ddd2721d8fea27d4967e7"
server_key_iv: "4086d59dc8ed26036a0752572448080b"
server_key_version: "encryptionkey@0"
I20260925 10:39:38.924345 28680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:39:38.929554 28680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:39:38.930225 28680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:39:38.930532 28680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:39:38.931300 28680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:39:38.932567 28680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:39:38.932647 28680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:38.932714 28680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:39:38.932760 28680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:38.964624 28680 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.1:44903
I20260925 10:39:38.964695 28806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.1:44903 every 8 connection(s)
I20260925 10:39:38.965854 28680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
I20260925 10:39:38.969694 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 28680
I20260925 10:39:38.969895 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/ts-0/wal/instance
I20260925 10:39:38.970585 23952 external_mini_cluster.cc:1428] Setting key 7e5a674edb57f70d0bf2d48857634dcd
I20260925 10:39:38.973255 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.2:0
--local_ip_for_outbound_sockets=127.23.100.2
--webserver_interface=127.23.100.2
--webserver_port=0
--tserver_master_addrs=127.23.100.62:35339
--builtin_ntp_servers=127.23.100.20:35399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 10:39:38.974629 28809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:38.968419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:35905 (local address 127.23.100.62:35339)
0925 10:39:38.968871 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:38.968877 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:38.970347 (+ 1470us) server_negotiation.cc:407] Connection header received
0925 10:39:38.972251 (+ 1904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:38.972256 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:38.972299 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:38.972381 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:38.973752 (+ 1371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:38.973761 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:38.973768 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:39:38.973791 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:38.973846 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:39:38.973863 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:38.974007 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:38.974164 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 10:39:38.974231 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":214,"threads_started":1}
I20260925 10:39:38.975390 28810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:38.969063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35339 (local address 127.23.100.1:35905)
0925 10:39:38.970167 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:38.970225 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:38.971985 (+ 1760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:38.972621 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:38.972629 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:38.972715 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:38.973439 (+ 724us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:39:38.973444 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:38.973495 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:39:38.973588 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:38.973972 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:38.973979 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:38.973985 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 10:39:38.974186 (+ 201us) client_negotiation.cc:240] Negotiation successful
0925 10:39:38.974460 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":370,"threads_started":1}
I20260925 10:39:38.976845 28807 heartbeater.cc:347] Connected to a master server at 127.23.100.62:35339
I20260925 10:39:38.977118 28807 heartbeater.cc:464] Registering TS with master...
I20260925 10:39:38.977650 28807 heartbeater.cc:511] Master 127.23.100.62:35339 requested a full tablet report, sending...
I20260925 10:39:38.979589 28621 ts_manager.cc:194] Registered new tserver with Master: f781e97a08174249a37b682c48d36189 (127.23.100.1:44903)
W20260925 10:39:39.287281 28811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:39:39.287523 28811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:39:39.287603 28811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:39:39.294410 28811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:39:39.294503 28811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:39:39.294615 28811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.2
I20260925 10:39:39.303290 28811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:35399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.2
--webserver_port=0
--tserver_master_addrs=127.23.100.62:35339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.2
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:39:39.304365 28811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:39:39.305917 28811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:39:39.318137 28819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:39.318132 28817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:39.318193 28816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:39.319020 28811 server_base.cc:1034] running on GCE node
I20260925 10:39:39.319602 28811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:39:39.320521 28811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:39:39.321830 28811 hybrid_clock.cc:648] HybridClock initialized: now 1790332779321785 us; error 59 us; skew 500 ppm
I20260925 10:39:39.322162 28811 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:39:39.324397 28811 webserver.cc:460] Webserver started at http://127.23.100.2:34795/ using document root <none> and password file <none>
I20260925 10:39:39.325048 28811 fs_manager.cc:359] Metadata directory not provided
I20260925 10:39:39.325176 28811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:39:39.325500 28811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:39:39.328315 28811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/ts-1/data/instance:
uuid: "41dd80bb58214c7e8c62b8f928505dc4"
format_stamp: "Formatted at 2026-09-25 10:39:39 on dist-test-slave-dxdd"
server_key: "8c083f606c3175dec29c33a64f94bb5e"
server_key_iv: "3219b7d37524b3b15ae51db25b2e24a7"
server_key_version: "encryptionkey@0"
I20260925 10:39:39.329043 28811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/ts-1/wal/instance:
uuid: "41dd80bb58214c7e8c62b8f928505dc4"
format_stamp: "Formatted at 2026-09-25 10:39:39 on dist-test-slave-dxdd"
server_key: "8c083f606c3175dec29c33a64f94bb5e"
server_key_iv: "3219b7d37524b3b15ae51db25b2e24a7"
server_key_version: "encryptionkey@0"
I20260925 10:39:39.333807 28811 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260925 10:39:39.336987 28825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:39.338193 28811 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 10:39:39.338415 28811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/ts-1/wal
uuid: "41dd80bb58214c7e8c62b8f928505dc4"
format_stamp: "Formatted at 2026-09-25 10:39:39 on dist-test-slave-dxdd"
server_key: "8c083f606c3175dec29c33a64f94bb5e"
server_key_iv: "3219b7d37524b3b15ae51db25b2e24a7"
server_key_version: "encryptionkey@0"
I20260925 10:39:39.338617 28811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:39:39.343734 28811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:39:39.344442 28811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:39:39.344684 28811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:39:39.345494 28811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:39:39.346843 28811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:39:39.346971 28811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:39.347100 28811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:39:39.347185 28811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:39.379366 28811 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.2:40021
I20260925 10:39:39.379451 28937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.2:40021 every 8 connection(s)
I20260925 10:39:39.380672 28811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
I20260925 10:39:39.383297 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 28811
I20260925 10:39:39.383473 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/ts-1/wal/instance
I20260925 10:39:39.383875 23952 external_mini_cluster.cc:1428] Setting key a622154a461b5ff4e8b6198c65be9174
I20260925 10:39:39.386524 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.3:0
--local_ip_for_outbound_sockets=127.23.100.3
--webserver_interface=127.23.100.3
--webserver_port=0
--tserver_master_addrs=127.23.100.62:35339
--builtin_ntp_servers=127.23.100.20:35399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 10:39:39.389191 28809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:39.383303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:46677 (local address 127.23.100.62:35339)
0925 10:39:39.383523 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:39.383529 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:39.385082 (+ 1553us) server_negotiation.cc:407] Connection header received
0925 10:39:39.386845 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:39.386850 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:39.386925 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:39.387029 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:39.388421 (+ 1392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:39.388428 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:39.388433 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:39:39.388459 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:39.388502 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:39:39.388514 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:39.388644 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:39.388859 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 10:39:39.388928 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 10:39:39.390120 28940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:39.383855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35339 (local address 127.23.100.2:46677)
0925 10:39:39.384921 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:39.384965 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:39.386525 (+ 1560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:39.387247 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:39.387254 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:39.387338 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:39.388047 (+ 709us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:39:39.388050 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:39.388086 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:39:39.388160 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:39.388617 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:39.388628 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:39.388636 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 10:39:39.388939 (+ 303us) client_negotiation.cc:240] Negotiation successful
0925 10:39:39.389207 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":344,"threads_started":1}
I20260925 10:39:39.391548 28938 heartbeater.cc:347] Connected to a master server at 127.23.100.62:35339
I20260925 10:39:39.391856 28938 heartbeater.cc:464] Registering TS with master...
I20260925 10:39:39.392590 28938 heartbeater.cc:511] Master 127.23.100.62:35339 requested a full tablet report, sending...
I20260925 10:39:39.394146 28621 ts_manager.cc:194] Registered new tserver with Master: 41dd80bb58214c7e8c62b8f928505dc4 (127.23.100.2:40021)
W20260925 10:39:39.701840 28941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:39:39.702087 28941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:39:39.702183 28941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:39:39.709342 28941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:39:39.709429 28941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:39:39.709554 28941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.3
I20260925 10:39:39.718034 28941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:35399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.3
--webserver_port=0
--tserver_master_addrs=127.23.100.62:35339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.3
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:39:39.719228 28941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:39:39.720893 28941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:39:39.733814 28946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:39.733863 28949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:39.733886 28947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:39.734786 28941 server_base.cc:1034] running on GCE node
I20260925 10:39:39.735284 28941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:39:39.736096 28941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:39:39.737285 28941 hybrid_clock.cc:648] HybridClock initialized: now 1790332779737235 us; error 48 us; skew 500 ppm
I20260925 10:39:39.737599 28941 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:39:39.739758 28941 webserver.cc:460] Webserver started at http://127.23.100.3:40197/ using document root <none> and password file <none>
I20260925 10:39:39.740389 28941 fs_manager.cc:359] Metadata directory not provided
I20260925 10:39:39.740528 28941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:39:39.740850 28941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:39:39.743654 28941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/ts-2/data/instance:
uuid: "19652bc7aa2a411db17a263f61f51b7f"
format_stamp: "Formatted at 2026-09-25 10:39:39 on dist-test-slave-dxdd"
server_key: "1b34dd7fb9ecce364e027eff6bef59a0"
server_key_iv: "f656a269d7ef45a774f5b3c64b073991"
server_key_version: "encryptionkey@0"
I20260925 10:39:39.744428 28941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/ts-2/wal/instance:
uuid: "19652bc7aa2a411db17a263f61f51b7f"
format_stamp: "Formatted at 2026-09-25 10:39:39 on dist-test-slave-dxdd"
server_key: "1b34dd7fb9ecce364e027eff6bef59a0"
server_key_iv: "f656a269d7ef45a774f5b3c64b073991"
server_key_version: "encryptionkey@0"
I20260925 10:39:39.749006 28941 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 10:39:39.752218 28955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:39.753429 28941 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 10:39:39.753577 28941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/ts-2/wal
uuid: "19652bc7aa2a411db17a263f61f51b7f"
format_stamp: "Formatted at 2026-09-25 10:39:39 on dist-test-slave-dxdd"
server_key: "1b34dd7fb9ecce364e027eff6bef59a0"
server_key_iv: "f656a269d7ef45a774f5b3c64b073991"
server_key_version: "encryptionkey@0"
I20260925 10:39:39.753780 28941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:39:39.758972 28941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:39:39.759701 28941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:39:39.759958 28941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:39:39.760874 28941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:39:39.762205 28941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:39:39.762313 28941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:39.762503 28941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:39:39.762594 28941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:39.792289 28941 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.3:45299
I20260925 10:39:39.792336 29067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.3:45299 every 8 connection(s)
I20260925 10:39:39.793545 28941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
I20260925 10:39:39.795892 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 28941
I20260925 10:39:39.796069 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/ts-2/wal/instance
I20260925 10:39:39.796619 23952 external_mini_cluster.cc:1428] Setting key 311ef75593c6e41c642854d541c5738a
I20260925 10:39:39.802063 28809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:39.796041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:34989 (local address 127.23.100.62:35339)
0925 10:39:39.796201 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:39.796205 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:39.797964 (+ 1759us) server_negotiation.cc:407] Connection header received
0925 10:39:39.800199 (+ 2235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:39.800203 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:39.800253 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:39.800329 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:39.801521 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:39.801524 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:39.801527 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:39:39.801540 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:39.801566 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:39:39.801576 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:39.801658 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:39.801803 (+ 145us) server_negotiation.cc:299] Negotiation successful
0925 10:39:39.801850 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 10:39:39.802977 29070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:39.796734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35339 (local address 127.23.100.3:34989)
0925 10:39:39.797807 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:39.797860 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:39.799976 (+ 2116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:39.800520 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:39.800527 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:39.800610 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:39.801278 (+ 668us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:39:39.801281 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:39.801314 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:39:39.801392 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:39.801649 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:39.801653 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:39.801658 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:39:39.801845 (+ 187us) client_negotiation.cc:240] Negotiation successful
0925 10:39:39.802020 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":339,"threads_started":1}
I20260925 10:39:39.804105 29068 heartbeater.cc:347] Connected to a master server at 127.23.100.62:35339
I20260925 10:39:39.804360 29068 heartbeater.cc:464] Registering TS with master...
I20260925 10:39:39.804906 29068 heartbeater.cc:511] Master 127.23.100.62:35339 requested a full tablet report, sending...
I20260925 10:39:39.806147 28621 ts_manager.cc:194] Registered new tserver with Master: 19652bc7aa2a411db17a263f61f51b7f (127.23.100.3:45299)
I20260925 10:39:39.811605 23952 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:39:39.817878 28809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:39.815775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33156 (local address 127.23.100.62:35339)
0925 10:39:39.815940 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:39.815944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:39.816452 (+ 508us) server_negotiation.cc:407] Connection header received
0925 10:39:39.816568 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:39.816572 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:39.816616 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:39.816692 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:39.817480 (+ 788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:39.817484 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:39.817487 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:39:39.817500 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:39.817528 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:39:39.817538 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:39.817629 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:39.817660 (+ 31us) server_negotiation.cc:299] Negotiation successful
0925 10:39:39.817700 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 10:39:39.821399 28621 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33156:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:39:39.828125 28621 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:39:39.858399 29082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:39.838871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.3:45299 (local address 127.0.0.1:37148)
0925 10:39:39.849968 (+ 11097us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:39.849989 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:39.850103 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:39.853952 (+ 3849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:39.853963 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:39.854624 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:39.855634 (+ 1010us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:39:39.855640 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:39.855680 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:39:39.855788 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:39.856992 (+ 1204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:39.857003 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:39.857010 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 10:39:39.857250 (+ 240us) client_negotiation.cc:240] Negotiation successful
0925 10:39:39.857521 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10915,"thread_start_us":10797,"threads_started":1}
I20260925 10:39:39.858527 29083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:39.839151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37148 (local address 127.23.100.3:45299)
0925 10:39:39.850760 (+ 11609us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:39.850773 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:39.850815 (+ 42us) server_negotiation.cc:407] Connection header received
0925 10:39:39.850901 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:39.850912 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:39.851205 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:39.851379 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:39.856628 (+ 5249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:39.856637 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:39.856650 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:39:39.856668 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:39.856721 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:39:39.856739 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:39.856947 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:39.857808 (+ 861us) server_negotiation.cc:299] Negotiation successful
0925 10:39:39.858073 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11386,"thread_start_us":14452,"threads_started":1}
I20260925 10:39:39.859686 29081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:39.838885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40132 (local address 127.23.100.2:40021)
0925 10:39:39.856399 (+ 17514us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:39.856417 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:39.856449 (+ 32us) server_negotiation.cc:407] Connection header received
0925 10:39:39.856522 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:39.856530 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:39.856743 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:39.857801 (+ 1058us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:39.858885 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:39.858919 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:39.858932 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:39:39.858950 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:39.859017 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:39:39.859038 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:39.859201 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:39.859262 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 10:39:39.859348 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17352,"thread_start_us":17183,"threads_started":1}
I20260925 10:39:39.859760 29084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:39.838579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.2:40021 (local address 127.0.0.1:40132)
0925 10:39:39.849448 (+ 10869us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:39.849510 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:39.849685 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:39.856974 (+ 7289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:39.856981 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:39.857014 (+ 33us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:39.857845 (+ 831us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:39:39.857848 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:39.857865 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:39:39.857912 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:39.859157 (+ 1245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:39.859162 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:39.859167 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:39:39.859290 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 10:39:39.859337 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10537,"thread_start_us":19047,"threads_started":1}
I20260925 10:39:39.861838 29085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:39.840794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53380 (local address 127.23.100.1:44903)
0925 10:39:39.857455 (+ 16661us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:39.857466 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:39.858494 (+ 1028us) server_negotiation.cc:407] Connection header received
0925 10:39:39.858709 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:39.858716 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:39.858937 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:39.859063 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:39.860921 (+ 1858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:39.860928 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:39.860939 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:39:39.860959 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:39.861014 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:39:39.861038 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:39.861150 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:39.861438 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 10:39:39.861533 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16282,"thread_start_us":16438,"threads_started":1}
I20260925 10:39:39.861924 29080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:39.840180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:44903 (local address 127.0.0.1:53380)
0925 10:39:39.858370 (+ 18190us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:39.858394 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:39.858580 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:39.859935 (+ 1355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:39.859939 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:39.859963 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:39.860688 (+ 725us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:39:39.860691 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:39.860706 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:39:39.860746 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:39.861267 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:39.861275 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:39.861281 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 10:39:39.861456 (+ 175us) client_negotiation.cc:240] Negotiation successful
0925 10:39:39.861503 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17969,"thread_start_us":8767,"threads_started":1}
I20260925 10:39:39.862560 29003 tablet_service.cc:1467] Processing CreateTablet for tablet 0a302fa85bf942e0a20bd3e94fc13ce6 (DEFAULT_TABLE table=test-table [id=d490047ff05a415fbf865dd6be7a0ed9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:39:39.863732 29003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a302fa85bf942e0a20bd3e94fc13ce6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:39:39.864723 28742 tablet_service.cc:1467] Processing CreateTablet for tablet 0a302fa85bf942e0a20bd3e94fc13ce6 (DEFAULT_TABLE table=test-table [id=d490047ff05a415fbf865dd6be7a0ed9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:39:39.865885 28742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a302fa85bf942e0a20bd3e94fc13ce6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:39:39.867756 28873 tablet_service.cc:1467] Processing CreateTablet for tablet 0a302fa85bf942e0a20bd3e94fc13ce6 (DEFAULT_TABLE table=test-table [id=d490047ff05a415fbf865dd6be7a0ed9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:39:39.869035 28873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a302fa85bf942e0a20bd3e94fc13ce6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:39:39.876561 29086 tablet_bootstrap.cc:492] T 0a302fa85bf942e0a20bd3e94fc13ce6 P f781e97a08174249a37b682c48d36189: Bootstrap starting.
I20260925 10:39:39.880424 29086 tablet_bootstrap.cc:654] T 0a302fa85bf942e0a20bd3e94fc13ce6 P f781e97a08174249a37b682c48d36189: Neither blocks nor log segments found. Creating new log.
I20260925 10:39:39.881397 29086 log.cc:824] T 0a302fa85bf942e0a20bd3e94fc13ce6 P f781e97a08174249a37b682c48d36189: Log is configured to *not* fsync() on all Append() calls
I20260925 10:39:39.883256 29087 tablet_bootstrap.cc:492] T 0a302fa85bf942e0a20bd3e94fc13ce6 P 19652bc7aa2a411db17a263f61f51b7f: Bootstrap starting.
I20260925 10:39:39.884269 29086 tablet_bootstrap.cc:492] T 0a302fa85bf942e0a20bd3e94fc13ce6 P f781e97a08174249a37b682c48d36189: No bootstrap required, opened a new log
I20260925 10:39:39.884539 29086 ts_tablet_manager.cc:1397] T 0a302fa85bf942e0a20bd3e94fc13ce6 P f781e97a08174249a37b682c48d36189: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260925 10:39:39.887257 29089 tablet_bootstrap.cc:492] T 0a302fa85bf942e0a20bd3e94fc13ce6 P 41dd80bb58214c7e8c62b8f928505dc4: Bootstrap starting.
I20260925 10:39:39.887902 29087 tablet_bootstrap.cc:654] T 0a302fa85bf942e0a20bd3e94fc13ce6 P 19652bc7aa2a411db17a263f61f51b7f: Neither blocks nor log segments found. Creating new log.
I20260925 10:39:39.888818 29087 log.cc:824] T 0a302fa85bf942e0a20bd3e94fc13ce6 P 19652bc7aa2a411db17a263f61f51b7f: Log is configured to *not* fsync() on all Append() calls
I20260925 10:39:39.890776 29089 tablet_bootstrap.cc:654] T 0a302fa85bf942e0a20bd3e94fc13ce6 P 41dd80bb58214c7e8c62b8f928505dc4: Neither blocks nor log segments found. Creating new log.
I20260925 10:39:39.891211 29087 tablet_bootstrap.cc:492] T 0a302fa85bf942e0a20bd3e94fc13ce6 P 19652bc7aa2a411db17a263f61f51b7f: No bootstrap required, opened a new log
I20260925 10:39:39.891445 29087 ts_tablet_manager.cc:1397] T 0a302fa85bf942e0a20bd3e94fc13ce6 P 19652bc7aa2a411db17a263f61f51b7f: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260925 10:39:39.891891 29089 log.cc:824] T 0a302fa85bf942e0a20bd3e94fc13ce6 P 41dd80bb58214c7e8c62b8f928505dc4: Log is configured to *not* fsync() on all Append() calls
I20260925 10:39:39.893774 29086 raft_consensus.cc:348] T 0a302fa85bf942e0a20bd3e94fc13ce6 P f781e97a08174249a37b682c48d36189 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41dd80bb58214c7e8c62b8f928505dc4" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 40021 } } peers { permanent_uuid: "19652bc7aa2a411db17a263f61f51b7f" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 45299 } } peers { permanent_uuid: "f781e97a08174249a37b682c48d36189" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 44903 } }
I20260925 10:39:39.894212 29086 raft_consensus.cc:374] T 0a302fa85bf942e0a20bd3e94fc13ce6 P f781e97a08174249a37b682c48d36189 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:39:39.894375 29086 raft_consensus.cc:729] T 0a302fa85bf942e0a20bd3e94fc13ce6 P f781e97a08174249a37b682c48d36189 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f781e97a08174249a37b682c48d36189, State: Initialized, Role: FOLLOWER
I20260925 10:39:39.894531 29089 tablet_bootstrap.cc:492] T 0a302fa85bf942e0a20bd3e94fc13ce6 P 41dd80bb58214c7e8c62b8f928505dc4: No bootstrap required, opened a new log
I20260925 10:39:39.894763 29089 ts_tablet_manager.cc:1397] T 0a302fa85bf942e0a20bd3e94fc13ce6 P 41dd80bb58214c7e8c62b8f928505dc4: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260925 10:39:39.894999 29086 consensus_queue.cc:260] T 0a302fa85bf942e0a20bd3e94fc13ce6 P f781e97a08174249a37b682c48d36189 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41dd80bb58214c7e8c62b8f928505dc4" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 40021 } } peers { permanent_uuid: "19652bc7aa2a411db17a263f61f51b7f" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 45299 } } peers { permanent_uuid: "f781e97a08174249a37b682c48d36189" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 44903 } }
I20260925 10:39:39.896054 28807 heartbeater.cc:503] Master 127.23.100.62:35339 was elected leader, sending a full tablet report...
I20260925 10:39:39.896656 29086 ts_tablet_manager.cc:1428] T 0a302fa85bf942e0a20bd3e94fc13ce6 P f781e97a08174249a37b682c48d36189: Time spent starting tablet: real 0.012s user 0.012s sys 0.001s
I20260925 10:39:39.901664 29087 raft_consensus.cc:348] T 0a302fa85bf942e0a20bd3e94fc13ce6 P 19652bc7aa2a411db17a263f61f51b7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41dd80bb58214c7e8c62b8f928505dc4" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 40021 } } peers { permanent_uuid: "19652bc7aa2a411db17a263f61f51b7f" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 45299 } } peers { permanent_uuid: "f781e97a08174249a37b682c48d36189" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 44903 } }
I20260925 10:39:39.902089 29087 raft_consensus.cc:374] T 0a302fa85bf942e0a20bd3e94fc13ce6 P 19652bc7aa2a411db17a263f61f51b7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:39:39.902189 29087 raft_consensus.cc:729] T 0a302fa85bf942e0a20bd3e94fc13ce6 P 19652bc7aa2a411db17a263f61f51b7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19652bc7aa2a411db17a263f61f51b7f, State: Initialized, Role: FOLLOWER
I20260925 10:39:39.902796 29087 consensus_queue.cc:260] T 0a302fa85bf942e0a20bd3e94fc13ce6 P 19652bc7aa2a411db17a263f61f51b7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41dd80bb58214c7e8c62b8f928505dc4" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 40021 } } peers { permanent_uuid: "19652bc7aa2a411db17a263f61f51b7f" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 45299 } } peers { permanent_uuid: "f781e97a08174249a37b682c48d36189" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 44903 } }
I20260925 10:39:39.903793 29068 heartbeater.cc:503] Master 127.23.100.62:35339 was elected leader, sending a full tablet report...
I20260925 10:39:39.904403 29087 ts_tablet_manager.cc:1428] T 0a302fa85bf942e0a20bd3e94fc13ce6 P 19652bc7aa2a411db17a263f61f51b7f: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260925 10:39:39.905560 29089 raft_consensus.cc:348] T 0a302fa85bf942e0a20bd3e94fc13ce6 P 41dd80bb58214c7e8c62b8f928505dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41dd80bb58214c7e8c62b8f928505dc4" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 40021 } } peers { permanent_uuid: "19652bc7aa2a411db17a263f61f51b7f" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 45299 } } peers { permanent_uuid: "f781e97a08174249a37b682c48d36189" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 44903 } }
I20260925 10:39:39.905941 29089 raft_consensus.cc:374] T 0a302fa85bf942e0a20bd3e94fc13ce6 P 41dd80bb58214c7e8c62b8f928505dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:39:39.906019 29089 raft_consensus.cc:729] T 0a302fa85bf942e0a20bd3e94fc13ce6 P 41dd80bb58214c7e8c62b8f928505dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41dd80bb58214c7e8c62b8f928505dc4, State: Initialized, Role: FOLLOWER
I20260925 10:39:39.906580 29089 consensus_queue.cc:260] T 0a302fa85bf942e0a20bd3e94fc13ce6 P 41dd80bb58214c7e8c62b8f928505dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41dd80bb58214c7e8c62b8f928505dc4" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 40021 } } peers { permanent_uuid: "19652bc7aa2a411db17a263f61f51b7f" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 45299 } } peers { permanent_uuid: "f781e97a08174249a37b682c48d36189" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 44903 } }
I20260925 10:39:39.907495 28938 heartbeater.cc:503] Master 127.23.100.62:35339 was elected leader, sending a full tablet report...
I20260925 10:39:39.908124 29089 ts_tablet_manager.cc:1428] T 0a302fa85bf942e0a20bd3e94fc13ce6 P 41dd80bb58214c7e8c62b8f928505dc4: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s
W20260925 10:39:39.968163 28808 tablet.cc:2369] T 0a302fa85bf942e0a20bd3e94fc13ce6 P f781e97a08174249a37b682c48d36189: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:39:40.045413 29069 tablet.cc:2369] T 0a302fa85bf942e0a20bd3e94fc13ce6 P 19652bc7aa2a411db17a263f61f51b7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:39:40.132665 28939 tablet.cc:2369] T 0a302fa85bf942e0a20bd3e94fc13ce6 P 41dd80bb58214c7e8c62b8f928505dc4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:39:40.179126 29093 raft_consensus.cc:482] T 0a302fa85bf942e0a20bd3e94fc13ce6 P 19652bc7aa2a411db17a263f61f51b7f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:39:40.179432 29093 raft_consensus.cc:504] T 0a302fa85bf942e0a20bd3e94fc13ce6 P 19652bc7aa2a411db17a263f61f51b7f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41dd80bb58214c7e8c62b8f928505dc4" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 40021 } } peers { permanent_uuid: "19652bc7aa2a411db17a263f61f51b7f" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 45299 } } peers { permanent_uuid: "f781e97a08174249a37b682c48d36189" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 44903 } }
I20260925 10:39:40.181073 29093 leader_election.cc:288] T 0a302fa85bf942e0a20bd3e94fc13ce6 P 19652bc7aa2a411db17a263f61f51b7f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41dd80bb58214c7e8c62b8f928505dc4 (127.23.100.2:40021), f781e97a08174249a37b682c48d36189 (127.23.100.1:44903)
I20260925 10:39:40.183602 29070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:40.181349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:44903 (local address 127.23.100.3:42121)
0925 10:39:40.181543 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:40.181563 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:40.181679 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:40.182060 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:40.182065 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:40.182095 (+ 30us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:40.182745 (+ 650us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:39:40.182750 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:40.182770 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:39:40.182817 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:40.183151 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:40.183161 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:40.183167 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 10:39:40.183299 (+ 132us) client_negotiation.cc:240] Negotiation successful
0925 10:39:40.183357 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 10:39:40.183619 29085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:40.181525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:42121 (local address 127.23.100.1:44903)
0925 10:39:40.181693 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:40.181701 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:40.181726 (+ 25us) server_negotiation.cc:407] Connection header received
0925 10:39:40.181801 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:40.181810 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:40.181901 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:40.182011 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:40.182965 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:40.182971 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:40.182978 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:39:40.182999 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:40.183034 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:39:40.183051 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:40.183161 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:40.183290 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 10:39:40.183335 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 10:39:40.184571 29081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:40.181496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:47947 (local address 127.23.100.2:40021)
0925 10:39:40.181661 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:40.181667 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:40.182367 (+ 700us) server_negotiation.cc:407] Connection header received
0925 10:39:40.182840 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:40.182844 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:40.182917 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:40.183017 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:40.184057 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:40.184063 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:40.184067 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:39:40.184090 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:40.184120 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:39:40.184131 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:40.184228 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:40.184347 (+ 119us) server_negotiation.cc:299] Negotiation successful
0925 10:39:40.184392 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 10:39:40.184530 28762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a302fa85bf942e0a20bd3e94fc13ce6" candidate_uuid: "19652bc7aa2a411db17a263f61f51b7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f781e97a08174249a37b682c48d36189" is_pre_election: true
I20260925 10:39:40.184803 29095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:40.181349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.2:40021 (local address 127.23.100.3:47947)
0925 10:39:40.182230 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:40.182247 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:40.182706 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:40.183042 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:40.183045 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:40.183060 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:40.183849 (+ 789us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:39:40.183852 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:40.183865 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:39:40.183901 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:40.184233 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:40.184239 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:40.184243 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:39:40.184352 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 10:39:40.184401 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":763,"mutex_wait_us":236,"thread_start_us":297,"threads_started":1}
I20260925 10:39:40.185078 28762 raft_consensus.cc:2454] T 0a302fa85bf942e0a20bd3e94fc13ce6 P f781e97a08174249a37b682c48d36189 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19652bc7aa2a411db17a263f61f51b7f in term 0.
I20260925 10:39:40.185714 28957 leader_election.cc:302] T 0a302fa85bf942e0a20bd3e94fc13ce6 P 19652bc7aa2a411db17a263f61f51b7f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19652bc7aa2a411db17a263f61f51b7f, f781e97a08174249a37b682c48d36189; no voters:
I20260925 10:39:40.185673 28893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a302fa85bf942e0a20bd3e94fc13ce6" candidate_uuid: "19652bc7aa2a411db17a263f61f51b7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41dd80bb58214c7e8c62b8f928505dc4" is_pre_election: true
I20260925 10:39:40.186072 29093 raft_consensus.cc:2781] T 0a302fa85bf942e0a20bd3e94fc13ce6 P 19652bc7aa2a411db17a263f61f51b7f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:39:40.186081 28893 raft_consensus.cc:2454] T 0a302fa85bf942e0a20bd3e94fc13ce6 P 41dd80bb58214c7e8c62b8f928505dc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19652bc7aa2a411db17a263f61f51b7f in term 0.
I20260925 10:39:40.186177 29093 raft_consensus.cc:482] T 0a302fa85bf942e0a20bd3e94fc13ce6 P 19652bc7aa2a411db17a263f61f51b7f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:39:40.186295 29093 raft_consensus.cc:3037] T 0a302fa85bf942e0a20bd3e94fc13ce6 P 19652bc7aa2a411db17a263f61f51b7f [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:39:40.188479 29093 raft_consensus.cc:504] T 0a302fa85bf942e0a20bd3e94fc13ce6 P 19652bc7aa2a411db17a263f61f51b7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41dd80bb58214c7e8c62b8f928505dc4" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 40021 } } peers { permanent_uuid: "19652bc7aa2a411db17a263f61f51b7f" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 45299 } } peers { permanent_uuid: "f781e97a08174249a37b682c48d36189" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 44903 } }
I20260925 10:39:40.189198 29093 leader_election.cc:288] T 0a302fa85bf942e0a20bd3e94fc13ce6 P 19652bc7aa2a411db17a263f61f51b7f [CANDIDATE]: Term 1 election: Requested vote from peers 41dd80bb58214c7e8c62b8f928505dc4 (127.23.100.2:40021), f781e97a08174249a37b682c48d36189 (127.23.100.1:44903)
I20260925 10:39:40.189616 28893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a302fa85bf942e0a20bd3e94fc13ce6" candidate_uuid: "19652bc7aa2a411db17a263f61f51b7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41dd80bb58214c7e8c62b8f928505dc4"
I20260925 10:39:40.189616 28762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a302fa85bf942e0a20bd3e94fc13ce6" candidate_uuid: "19652bc7aa2a411db17a263f61f51b7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f781e97a08174249a37b682c48d36189"
I20260925 10:39:40.189819 28893 raft_consensus.cc:3037] T 0a302fa85bf942e0a20bd3e94fc13ce6 P 41dd80bb58214c7e8c62b8f928505dc4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:39:40.189819 28762 raft_consensus.cc:3037] T 0a302fa85bf942e0a20bd3e94fc13ce6 P f781e97a08174249a37b682c48d36189 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:39:40.193055 28762 raft_consensus.cc:2454] T 0a302fa85bf942e0a20bd3e94fc13ce6 P f781e97a08174249a37b682c48d36189 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19652bc7aa2a411db17a263f61f51b7f in term 1.
I20260925 10:39:40.193055 28893 raft_consensus.cc:2454] T 0a302fa85bf942e0a20bd3e94fc13ce6 P 41dd80bb58214c7e8c62b8f928505dc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19652bc7aa2a411db17a263f61f51b7f in term 1.
I20260925 10:39:40.193418 28957 leader_election.cc:302] T 0a302fa85bf942e0a20bd3e94fc13ce6 P 19652bc7aa2a411db17a263f61f51b7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19652bc7aa2a411db17a263f61f51b7f, f781e97a08174249a37b682c48d36189; no voters:
I20260925 10:39:40.193678 29093 raft_consensus.cc:2781] T 0a302fa85bf942e0a20bd3e94fc13ce6 P 19652bc7aa2a411db17a263f61f51b7f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:39:40.194088 29093 raft_consensus.cc:686] T 0a302fa85bf942e0a20bd3e94fc13ce6 P 19652bc7aa2a411db17a263f61f51b7f [term 1 LEADER]: Becoming Leader. State: Replica: 19652bc7aa2a411db17a263f61f51b7f, State: Running, Role: LEADER
I20260925 10:39:40.194581 29093 consensus_queue.cc:237] T 0a302fa85bf942e0a20bd3e94fc13ce6 P 19652bc7aa2a411db17a263f61f51b7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41dd80bb58214c7e8c62b8f928505dc4" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 40021 } } peers { permanent_uuid: "19652bc7aa2a411db17a263f61f51b7f" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 45299 } } peers { permanent_uuid: "f781e97a08174249a37b682c48d36189" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 44903 } }
I20260925 10:39:40.199277 28621 catalog_manager.cc:5495] T 0a302fa85bf942e0a20bd3e94fc13ce6 P 19652bc7aa2a411db17a263f61f51b7f reported cstate change: term changed from 0 to 1, leader changed from <none> to 19652bc7aa2a411db17a263f61f51b7f (127.23.100.3). New cstate: current_term: 1 leader_uuid: "19652bc7aa2a411db17a263f61f51b7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41dd80bb58214c7e8c62b8f928505dc4" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 40021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19652bc7aa2a411db17a263f61f51b7f" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 45299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f781e97a08174249a37b682c48d36189" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 44903 } health_report { overall_health: UNKNOWN } } }
I20260925 10:39:40.218750 29083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:40.216908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37156 (local address 127.23.100.3:45299)
0925 10:39:40.217089 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:40.217093 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:40.217109 (+ 16us) server_negotiation.cc:407] Connection header received
0925 10:39:40.217278 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:40.217282 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:40.217345 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:40.217415 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:40.218122 (+ 707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:40.218126 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:40.218128 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:39:40.218141 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:40.218167 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:39:40.218177 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:40.218265 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:40.218437 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 10:39:40.218484 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 10:39:40.225991 28762 raft_consensus.cc:1261] T 0a302fa85bf942e0a20bd3e94fc13ce6 P f781e97a08174249a37b682c48d36189 [term 1 FOLLOWER]: Refusing update from remote peer 19652bc7aa2a411db17a263f61f51b7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:39:40.226013 28893 raft_consensus.cc:1261] T 0a302fa85bf942e0a20bd3e94fc13ce6 P 41dd80bb58214c7e8c62b8f928505dc4 [term 1 FOLLOWER]: Refusing update from remote peer 19652bc7aa2a411db17a263f61f51b7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:39:40.226930 29096 consensus_queue.cc:1035] T 0a302fa85bf942e0a20bd3e94fc13ce6 P 19652bc7aa2a411db17a263f61f51b7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "41dd80bb58214c7e8c62b8f928505dc4" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 40021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:39:40.227245 29093 consensus_queue.cc:1035] T 0a302fa85bf942e0a20bd3e94fc13ce6 P 19652bc7aa2a411db17a263f61f51b7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f781e97a08174249a37b682c48d36189" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:39:40.239059 29099 mvcc.cc:205] Tried to move back new op lower bound from 7333203067796672512 to 7333203067680301056. Current Snapshot: MvccSnapshot[applied={T|T < 7333203067796672512}]
I20260925 10:39:40.252369 28620 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:39:40.252614 28620 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:39:40.256681 28620 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d063adba002c4890ab69c4aff9b886ae: Sending DeleteTablet for 3 replicas of tablet 0a302fa85bf942e0a20bd3e94fc13ce6
I20260925 10:39:40.258796 28873 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a302fa85bf942e0a20bd3e94fc13ce6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:39:40 UTC) from {username='slave'} at 127.0.0.1:40132
I20260925 10:39:40.258982 29003 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a302fa85bf942e0a20bd3e94fc13ce6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:39:40 UTC) from {username='slave'} at 127.0.0.1:37148
I20260925 10:39:40.259660 29109 tablet_replica.cc:330] T 0a302fa85bf942e0a20bd3e94fc13ce6 P 41dd80bb58214c7e8c62b8f928505dc4: stopping tablet replica
I20260925 10:39:40.260089 29109 raft_consensus.cc:2229] T 0a302fa85bf942e0a20bd3e94fc13ce6 P 41dd80bb58214c7e8c62b8f928505dc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:39:40.260191 28742 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a302fa85bf942e0a20bd3e94fc13ce6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:39:40 UTC) from {username='slave'} at 127.0.0.1:53380
I20260925 10:39:40.260468 29110 tablet_replica.cc:330] T 0a302fa85bf942e0a20bd3e94fc13ce6 P 19652bc7aa2a411db17a263f61f51b7f: stopping tablet replica
I20260925 10:39:40.260663 29109 raft_consensus.cc:2258] T 0a302fa85bf942e0a20bd3e94fc13ce6 P 41dd80bb58214c7e8c62b8f928505dc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:39:40.260881 29110 raft_consensus.cc:2229] T 0a302fa85bf942e0a20bd3e94fc13ce6 P 19652bc7aa2a411db17a263f61f51b7f [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:39:40.261884 29110 raft_consensus.cc:2258] T 0a302fa85bf942e0a20bd3e94fc13ce6 P 19652bc7aa2a411db17a263f61f51b7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:39:40.262169 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 28680
I20260925 10:39:40.262778 29109 ts_tablet_manager.cc:1905] T 0a302fa85bf942e0a20bd3e94fc13ce6 P 41dd80bb58214c7e8c62b8f928505dc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:39:40.263890 29110 ts_tablet_manager.cc:1905] T 0a302fa85bf942e0a20bd3e94fc13ce6 P 19652bc7aa2a411db17a263f61f51b7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:39:40.269454 29109 ts_tablet_manager.cc:1918] T 0a302fa85bf942e0a20bd3e94fc13ce6 P 41dd80bb58214c7e8c62b8f928505dc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:39:40.269670 29109 log.cc:1196] T 0a302fa85bf942e0a20bd3e94fc13ce6 P 41dd80bb58214c7e8c62b8f928505dc4: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/ts-1/wal/wals/0a302fa85bf942e0a20bd3e94fc13ce6
I20260925 10:39:40.270207 29109 ts_tablet_manager.cc:1939] T 0a302fa85bf942e0a20bd3e94fc13ce6 P 41dd80bb58214c7e8c62b8f928505dc4: Deleting consensus metadata
I20260925 10:39:40.271970 28608 catalog_manager.cc:4841] TS 41dd80bb58214c7e8c62b8f928505dc4 (127.23.100.2:40021): tablet 0a302fa85bf942e0a20bd3e94fc13ce6 (table test-table [id=d490047ff05a415fbf865dd6be7a0ed9]) successfully deleted
I20260925 10:39:40.272229 29110 ts_tablet_manager.cc:1918] T 0a302fa85bf942e0a20bd3e94fc13ce6 P 19652bc7aa2a411db17a263f61f51b7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:39:40.272420 29110 log.cc:1196] T 0a302fa85bf942e0a20bd3e94fc13ce6 P 19652bc7aa2a411db17a263f61f51b7f: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790332740062128-23952-0/minicluster-data/ts-2/wal/wals/0a302fa85bf942e0a20bd3e94fc13ce6
I20260925 10:39:40.272967 29110 ts_tablet_manager.cc:1939] T 0a302fa85bf942e0a20bd3e94fc13ce6 P 19652bc7aa2a411db17a263f61f51b7f: Deleting consensus metadata
I20260925 10:39:40.274712 28605 catalog_manager.cc:4841] TS 19652bc7aa2a411db17a263f61f51b7f (127.23.100.3:45299): tablet 0a302fa85bf942e0a20bd3e94fc13ce6 (table test-table [id=d490047ff05a415fbf865dd6be7a0ed9]) successfully deleted
I20260925 10:39:40.279421 29111 tablet_replica.cc:330] T 0a302fa85bf942e0a20bd3e94fc13ce6 P f781e97a08174249a37b682c48d36189: stopping tablet replica
I20260925 10:39:40.279841 29111 raft_consensus.cc:2229] T 0a302fa85bf942e0a20bd3e94fc13ce6 P f781e97a08174249a37b682c48d36189 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:39:40.280314 29111 raft_consensus.cc:2258] T 0a302fa85bf942e0a20bd3e94fc13ce6 P f781e97a08174249a37b682c48d36189 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:39:40.281298 29085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:40.265148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53386 (local address 127.23.100.1:44903)
0925 10:39:40.265311 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:40.265316 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:40.278909 (+ 13593us) server_negotiation.cc:407] Connection header received
0925 10:39:40.279170 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:40.279178 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:40.279285 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:40.279347 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:40.280436 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:40.280441 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:40.280446 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:39:40.280464 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:40.280501 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:39:40.280512 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:40.280667 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:40.280950 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 10:39:40.281015 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 10:39:40.281968 28802 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:39:40.282539 29111 ts_tablet_manager.cc:1905] T 0a302fa85bf942e0a20bd3e94fc13ce6 P f781e97a08174249a37b682c48d36189: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:39:40.370405 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 28680
W20260925 10:39:40.394520 28606 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.100.1:44903 (error 108)
I20260925 10:39:40.394744 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 28811
I20260925 10:39:40.398762 29081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:40.395550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40144 (local address 127.23.100.2:40021)
0925 10:39:40.395787 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:40.395795 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:40.396145 (+ 350us) server_negotiation.cc:407] Connection header received
0925 10:39:40.396366 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:40.396374 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:40.396501 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:40.396609 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:40.397717 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:40.397724 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:40.397728 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:39:40.397747 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:40.397787 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:39:40.397803 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:40.397925 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:40.398356 (+ 431us) server_negotiation.cc:299] Negotiation successful
0925 10:39:40.398420 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 10:39:40.399714 28933 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:39:40.401126 29116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:40.399844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:44903 (local address 127.0.0.1:53396)
0925 10:39:40.400755 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:40.400895 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.1:44903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":794,"thread_start_us":677,"threads_started":1}
W20260925 10:39:40.401371 28606 catalog_manager.cc:4584] TS f781e97a08174249a37b682c48d36189 (127.23.100.1:44903): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0a302fa85bf942e0a20bd3e94fc13ce6: Network error: Client connection negotiation failed: client connection to 127.23.100.1:44903: connect: Connection refused (error 111)
I20260925 10:39:40.454962 29116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:40.454447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:44903 (local address 127.0.0.1:53400)
0925 10:39:40.454675 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:40.454806 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.1:44903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 10:39:40.456101 29116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:40.455691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:44903 (local address 127.0.0.1:53402)
0925 10:39:40.455878 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:40.455950 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.1:44903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260925 10:39:40.501116 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 28811
I20260925 10:39:40.516872 29116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:40.516295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:44903 (local address 127.0.0.1:53404)
0925 10:39:40.516549 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:40.516676 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.1:44903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260925 10:39:40.518107 29116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:40.517668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:44903 (local address 127.0.0.1:53414)
0925 10:39:40.517842 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:40.517935 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.1:44903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 10:39:40.518532 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 28941
I20260925 10:39:40.521365 29083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:40.519282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37162 (local address 127.23.100.3:45299)
0925 10:39:40.519501 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:40.519508 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:40.519566 (+ 58us) server_negotiation.cc:407] Connection header received
0925 10:39:40.519679 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:40.519685 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:40.519753 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:40.519854 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:40.520692 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:40.520698 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:40.520703 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:39:40.520721 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:40.520756 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:39:40.520774 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:40.520920 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:40.520984 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 10:39:40.521040 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 10:39:40.522171 29063 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:39:40.598729 29116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:40.598078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:44903 (local address 127.0.0.1:53418)
0925 10:39:40.598445 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:40.598565 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.1:44903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260925 10:39:40.599879 29116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:40.599426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:44903 (local address 127.0.0.1:53420)
0925 10:39:40.599669 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:40.599756 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.1:44903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260925 10:39:40.638485 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 28941
I20260925 10:39:40.657308 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 28590
I20260925 10:39:40.658548 28651 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:39:40.764168 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 28590
2026-09-25T10:39:40Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2800 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 10:39:40.804677 23952 mini_oidc.cc:152] Starting JWKS server
I20260925 10:39:40.807142 23952 webserver.cc:460] Webserver started at https://127.0.0.1:45309/ using document root <none> and password file <none>
I20260925 10:39:40.807446 23952 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 10:39:40.808040 23952 webserver.cc:460] Webserver started at http://127.0.0.1:46095/ using document root <none> and password file <none>
2026-09-25T10:39:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:39:40Z Disabled control of system clock
I20260925 10:39:40.834100 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:40941
--webserver_interface=127.23.100.62
--webserver_port=0
--builtin_ntp_servers=127.23.100.20:39167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.62:40941
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45309/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790332740062128-23952-0/testchainca.cert with env {}
W20260925 10:39:41.148744 29124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:39:41.149000 29124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:39:41.149082 29124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:39:41.155874 29124 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:39:41.155969 29124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:39:41.156042 29124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:39:41.156100 29124 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:39:41.156136 29124 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 10:39:41.156195 29124 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 10:39:41.156234 29124 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45309/jwks.json
I20260925 10:39:41.165530 29124 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:39167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790332740062128-23952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.62:40941
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:40941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45309/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790332740062128-23952-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:39:41.166698 29124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:39:41.167888 29124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:39:41.178921 29129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:41.179029 29132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:41.178958 29130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:41.179919 29124 server_base.cc:1034] running on GCE node
I20260925 10:39:41.180610 29124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:39:41.181813 29124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:39:41.183000 29124 hybrid_clock.cc:648] HybridClock initialized: now 1790332781182955 us; error 42 us; skew 500 ppm
I20260925 10:39:41.183281 29124 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:39:41.185250 29124 webserver.cc:460] Webserver started at http://127.23.100.62:34915/ using document root <none> and password file <none>
I20260925 10:39:41.185855 29124 fs_manager.cc:359] Metadata directory not provided
I20260925 10:39:41.185936 29124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:39:41.186174 29124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:39:41.188880 29124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790332740062128-23952-0/minicluster-data/master-0/data/instance:
uuid: "5811303b23f44f1eae8e04a2933706d3"
format_stamp: "Formatted at 2026-09-25 10:39:41 on dist-test-slave-dxdd"
server_key: "e69810d4b7ed63714b4325ce1dc688c7"
server_key_iv: "e7e63da2bf5f19e5224b7b6b4d407a42"
server_key_version: "encryptionkey@0"
I20260925 10:39:41.189575 29124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790332740062128-23952-0/minicluster-data/master-0/wal/instance:
uuid: "5811303b23f44f1eae8e04a2933706d3"
format_stamp: "Formatted at 2026-09-25 10:39:41 on dist-test-slave-dxdd"
server_key: "e69810d4b7ed63714b4325ce1dc688c7"
server_key_iv: "e7e63da2bf5f19e5224b7b6b4d407a42"
server_key_version: "encryptionkey@0"
I20260925 10:39:41.194448 29124 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 10:39:41.198529 29138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:41.200361 29124 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 10:39:41.200536 29124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790332740062128-23952-0/minicluster-data/master-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790332740062128-23952-0/minicluster-data/master-0/wal
uuid: "5811303b23f44f1eae8e04a2933706d3"
format_stamp: "Formatted at 2026-09-25 10:39:41 on dist-test-slave-dxdd"
server_key: "e69810d4b7ed63714b4325ce1dc688c7"
server_key_iv: "e7e63da2bf5f19e5224b7b6b4d407a42"
server_key_version: "encryptionkey@0"
I20260925 10:39:41.200762 29124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790332740062128-23952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790332740062128-23952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790332740062128-23952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:39:41.224443 29124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:39:41.225198 29124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:39:41.225489 29124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:39:41.242688 29124 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.62:40941
I20260925 10:39:41.242712 29189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.62:40941 every 8 connection(s)
I20260925 10:39:41.243873 29124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
I20260925 10:39:41.247300 29190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:39:41.250145 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 29124
I20260925 10:39:41.250396 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790332740062128-23952-0/minicluster-data/master-0/wal/instance
I20260925 10:39:41.251011 23952 external_mini_cluster.cc:1428] Setting key ccb23afe9dc7495b61690fe437eca2ed
I20260925 10:39:41.258713 29190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5811303b23f44f1eae8e04a2933706d3: Bootstrap starting.
I20260925 10:39:41.262445 29190 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5811303b23f44f1eae8e04a2933706d3: Neither blocks nor log segments found. Creating new log.
I20260925 10:39:41.263468 29190 log.cc:824] T 00000000000000000000000000000000 P 5811303b23f44f1eae8e04a2933706d3: Log is configured to *not* fsync() on all Append() calls
I20260925 10:39:41.264736 29193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:41.252755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59958 (local address 127.23.100.62:40941)
0925 10:39:41.253640 (+ 885us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:41.253649 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:41.253685 (+ 36us) server_negotiation.cc:407] Connection header received
0925 10:39:41.255343 (+ 1658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:41.255363 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:41.255817 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:41.256200 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:41.256775 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:41.258240 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:41.260155 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:41.261175 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:41.261375 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:41.262577 (+ 1202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:41.262587 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:41.262609 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:39:41.263128 (+ 519us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:41.263213 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:39:41.263236 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:41.263360 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:41.263545 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 10:39:41.263851 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":317,"threads_started":1}
I20260925 10:39:41.266065 29190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5811303b23f44f1eae8e04a2933706d3: No bootstrap required, opened a new log
I20260925 10:39:41.268240 23952 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:39:41.274489 29190 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5811303b23f44f1eae8e04a2933706d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5811303b23f44f1eae8e04a2933706d3" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 40941 } }
I20260925 10:39:41.274823 29190 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5811303b23f44f1eae8e04a2933706d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:39:41.274966 29190 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5811303b23f44f1eae8e04a2933706d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5811303b23f44f1eae8e04a2933706d3, State: Initialized, Role: FOLLOWER
I20260925 10:39:41.275545 29190 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5811303b23f44f1eae8e04a2933706d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5811303b23f44f1eae8e04a2933706d3" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 40941 } }
I20260925 10:39:41.275774 29190 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5811303b23f44f1eae8e04a2933706d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:39:41.275913 29190 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5811303b23f44f1eae8e04a2933706d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:39:41.276095 29190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5811303b23f44f1eae8e04a2933706d3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:39:41.278064 29190 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5811303b23f44f1eae8e04a2933706d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5811303b23f44f1eae8e04a2933706d3" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 40941 } }
I20260925 10:39:41.278638 29190 leader_election.cc:302] T 00000000000000000000000000000000 P 5811303b23f44f1eae8e04a2933706d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5811303b23f44f1eae8e04a2933706d3; no voters:
I20260925 10:39:41.279258 29190 leader_election.cc:288] T 00000000000000000000000000000000 P 5811303b23f44f1eae8e04a2933706d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:39:41.279402 29195 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5811303b23f44f1eae8e04a2933706d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:39:41.280118 29195 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5811303b23f44f1eae8e04a2933706d3 [term 1 LEADER]: Becoming Leader. State: Replica: 5811303b23f44f1eae8e04a2933706d3, State: Running, Role: LEADER
I20260925 10:39:41.280610 29195 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5811303b23f44f1eae8e04a2933706d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5811303b23f44f1eae8e04a2933706d3" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 40941 } }
I20260925 10:39:41.281159 29190 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5811303b23f44f1eae8e04a2933706d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:39:41.283993 29196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5811303b23f44f1eae8e04a2933706d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5811303b23f44f1eae8e04a2933706d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5811303b23f44f1eae8e04a2933706d3" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 40941 } } }
I20260925 10:39:41.283972 29197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5811303b23f44f1eae8e04a2933706d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5811303b23f44f1eae8e04a2933706d3. Latest consensus state: current_term: 1 leader_uuid: "5811303b23f44f1eae8e04a2933706d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5811303b23f44f1eae8e04a2933706d3" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 40941 } } }
I20260925 10:39:41.284441 29196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5811303b23f44f1eae8e04a2933706d3 [sys.catalog]: This master's current role is: LEADER
I20260925 10:39:41.284474 29197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5811303b23f44f1eae8e04a2933706d3 [sys.catalog]: This master's current role is: LEADER
I20260925 10:39:41.285997 29203 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:39:41.290630 29203 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:39:41.299633 29203 catalog_manager.cc:1347] Generated new cluster ID: 4dbdd3bb1a8444b28d34fea175469461
I20260925 10:39:41.299743 29203 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:39:41.318357 29203 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:39:41.319739 29203 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:39:41.329804 29203 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5811303b23f44f1eae8e04a2933706d3: Generated new TSK 0
I20260925 10:39:41.330791 29203 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 10:39:41.370538 29220 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 10:39:41.370718 29220 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 10:39:41.371848 29193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:41.350401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59968 (local address 127.23.100.62:40941)
0925 10:39:41.350629 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:41.350635 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:41.350998 (+ 363us) server_negotiation.cc:407] Connection header received
0925 10:39:41.351276 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:41.351282 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:41.351394 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:41.351559 (+ 165us) server_negotiation.cc:226] Negotiated authn=JWT
0925 10:39:41.352058 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:41.353173 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:41.354682 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:41.355428 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:41.355568 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:41.356060 (+ 492us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 10:39:41.371199 (+ 15139us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45309/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 10:39:41.371545 (+ 346us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59968: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45309/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":90}
W20260925 10:39:41.371982 29193 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59968: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45309/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260925 10:39:41.373142 29219 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:39:41.350200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:40941 (local address 127.0.0.1:59968)
0925 10:39:41.350894 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:41.350909 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:41.351052 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:41.351587 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:41.351590 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:41.351599 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0925 10:39:41.351924 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:41.351931 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:41.353324 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:41.353333 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:41.354548 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:41.354558 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:41.355608 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:41.355612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:41.355732 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:41.355783 (+ 51us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 10:39:41.371486 (+ 15703us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45309/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 10:39:41.371738 (+ 252us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.100.62:40941: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45309/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":385,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.23.100.62:40941: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45309/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 10:39:41.375234 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 29124
I20260925 10:39:41.376333 29185 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:39:41.467339 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 29124
2026-09-25T10:39:41Z chronyd exiting
I20260925 10:39:41.517283 23952 test_util.cc:175] -----------------------------------------------
I20260925 10:39:41.517436 23952 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790332740062128-23952-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (716 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 10:39:41.521011 23952 mini_oidc.cc:152] Starting JWKS server
I20260925 10:39:41.522850 23952 webserver.cc:460] Webserver started at https://127.0.0.1:42869/ using document root <none> and password file <none>
I20260925 10:39:41.523096 23952 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 10:39:41.523716 23952 webserver.cc:460] Webserver started at http://127.0.0.1:46095/ using document root <none> and password file <none>
2026-09-25T10:39:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:39:41Z Disabled control of system clock
I20260925 10:39:41.548395 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:43499
--webserver_interface=127.23.100.62
--webserver_port=0
--builtin_ntp_servers=127.23.100.20:41847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.62:43499
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42869/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790332740062128-23952-0/testchainca.cert with env {}
W20260925 10:39:41.857079 29227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:39:41.857339 29227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:39:41.857437 29227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:39:41.864439 29227 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:39:41.864538 29227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:39:41.864585 29227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:39:41.864634 29227 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:39:41.864679 29227 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 10:39:41.864732 29227 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 10:39:41.864773 29227 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42869/jwks.json
I20260925 10:39:41.874171 29227 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:41847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790332740062128-23952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.62:43499
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:43499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42869/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790332740062128-23952-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:39:41.875602 29227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:39:41.876869 29227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:39:41.888244 29232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:41.888284 29235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:41.888321 29233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:41.889111 29227 server_base.cc:1034] running on GCE node
I20260925 10:39:41.889901 29227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:39:41.891358 29227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:39:41.892628 29227 hybrid_clock.cc:648] HybridClock initialized: now 1790332781892585 us; error 41 us; skew 500 ppm
I20260925 10:39:41.893002 29227 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:39:41.895228 29227 webserver.cc:460] Webserver started at http://127.23.100.62:42183/ using document root <none> and password file <none>
I20260925 10:39:41.895915 29227 fs_manager.cc:359] Metadata directory not provided
I20260925 10:39:41.896055 29227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:39:41.896370 29227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:39:41.899214 29227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790332740062128-23952-0/minicluster-data/master-0/data/instance:
uuid: "571a89c8ff6d42db92c3bd101bae6c5f"
format_stamp: "Formatted at 2026-09-25 10:39:41 on dist-test-slave-dxdd"
server_key: "818c5abb84c8c382f501de676d900496"
server_key_iv: "4d9d13b2b9e486116e40eb2ed614019e"
server_key_version: "encryptionkey@0"
I20260925 10:39:41.899981 29227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790332740062128-23952-0/minicluster-data/master-0/wal/instance:
uuid: "571a89c8ff6d42db92c3bd101bae6c5f"
format_stamp: "Formatted at 2026-09-25 10:39:41 on dist-test-slave-dxdd"
server_key: "818c5abb84c8c382f501de676d900496"
server_key_iv: "4d9d13b2b9e486116e40eb2ed614019e"
server_key_version: "encryptionkey@0"
I20260925 10:39:41.904784 29227 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 10:39:41.907931 29241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:41.909124 29227 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 10:39:41.909333 29227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790332740062128-23952-0/minicluster-data/master-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790332740062128-23952-0/minicluster-data/master-0/wal
uuid: "571a89c8ff6d42db92c3bd101bae6c5f"
format_stamp: "Formatted at 2026-09-25 10:39:41 on dist-test-slave-dxdd"
server_key: "818c5abb84c8c382f501de676d900496"
server_key_iv: "4d9d13b2b9e486116e40eb2ed614019e"
server_key_version: "encryptionkey@0"
I20260925 10:39:41.909538 29227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790332740062128-23952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790332740062128-23952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790332740062128-23952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:39:41.940434 29227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:39:41.941288 29227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:39:41.941614 29227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:39:41.959599 29227 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.62:43499
I20260925 10:39:41.959618 29292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.62:43499 every 8 connection(s)
I20260925 10:39:41.960976 29227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
I20260925 10:39:41.964704 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 29227
I20260925 10:39:41.964825 29293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:39:41.964913 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790332740062128-23952-0/minicluster-data/master-0/wal/instance
I20260925 10:39:41.965793 23952 external_mini_cluster.cc:1428] Setting key aba67091aee2e9a8df2bf44d47ba2ebc
I20260925 10:39:41.977813 29293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 571a89c8ff6d42db92c3bd101bae6c5f: Bootstrap starting.
I20260925 10:39:41.980573 29293 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 571a89c8ff6d42db92c3bd101bae6c5f: Neither blocks nor log segments found. Creating new log.
I20260925 10:39:41.981448 29293 log.cc:824] T 00000000000000000000000000000000 P 571a89c8ff6d42db92c3bd101bae6c5f: Log is configured to *not* fsync() on all Append() calls
I20260925 10:39:41.984282 29293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 571a89c8ff6d42db92c3bd101bae6c5f: No bootstrap required, opened a new log
I20260925 10:39:41.984699 29296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:41.967744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60704 (local address 127.23.100.62:43499)
0925 10:39:41.968747 (+ 1003us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:41.968757 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:41.968799 (+ 42us) server_negotiation.cc:407] Connection header received
0925 10:39:41.970852 (+ 2053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:41.970873 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:41.971316 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:41.971647 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:41.972166 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:41.973392 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:41.975263 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:41.975927 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:41.976070 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:41.982445 (+ 6375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:41.982460 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:41.982478 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:39:41.983057 (+ 579us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:41.983132 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:39:41.983153 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:41.983238 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:41.983473 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 10:39:41.983799 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":403,"threads_started":1}
I20260925 10:39:41.988392 23952 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:39:41.992822 29293 raft_consensus.cc:348] T 00000000000000000000000000000000 P 571a89c8ff6d42db92c3bd101bae6c5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "571a89c8ff6d42db92c3bd101bae6c5f" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 43499 } }
I20260925 10:39:41.993124 29293 raft_consensus.cc:374] T 00000000000000000000000000000000 P 571a89c8ff6d42db92c3bd101bae6c5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:39:41.993206 29293 raft_consensus.cc:729] T 00000000000000000000000000000000 P 571a89c8ff6d42db92c3bd101bae6c5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 571a89c8ff6d42db92c3bd101bae6c5f, State: Initialized, Role: FOLLOWER
I20260925 10:39:41.993643 29293 consensus_queue.cc:260] T 00000000000000000000000000000000 P 571a89c8ff6d42db92c3bd101bae6c5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "571a89c8ff6d42db92c3bd101bae6c5f" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 43499 } }
I20260925 10:39:41.993832 29293 raft_consensus.cc:388] T 00000000000000000000000000000000 P 571a89c8ff6d42db92c3bd101bae6c5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:39:41.993923 29293 raft_consensus.cc:482] T 00000000000000000000000000000000 P 571a89c8ff6d42db92c3bd101bae6c5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:39:41.994029 29293 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 571a89c8ff6d42db92c3bd101bae6c5f [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:39:41.996093 29293 raft_consensus.cc:504] T 00000000000000000000000000000000 P 571a89c8ff6d42db92c3bd101bae6c5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "571a89c8ff6d42db92c3bd101bae6c5f" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 43499 } }
I20260925 10:39:41.996606 29293 leader_election.cc:302] T 00000000000000000000000000000000 P 571a89c8ff6d42db92c3bd101bae6c5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 571a89c8ff6d42db92c3bd101bae6c5f; no voters:
I20260925 10:39:41.997129 29293 leader_election.cc:288] T 00000000000000000000000000000000 P 571a89c8ff6d42db92c3bd101bae6c5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:39:41.997260 29298 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 571a89c8ff6d42db92c3bd101bae6c5f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:39:41.998006 29298 raft_consensus.cc:686] T 00000000000000000000000000000000 P 571a89c8ff6d42db92c3bd101bae6c5f [term 1 LEADER]: Becoming Leader. State: Replica: 571a89c8ff6d42db92c3bd101bae6c5f, State: Running, Role: LEADER
I20260925 10:39:41.998559 29298 consensus_queue.cc:237] T 00000000000000000000000000000000 P 571a89c8ff6d42db92c3bd101bae6c5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "571a89c8ff6d42db92c3bd101bae6c5f" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 43499 } }
I20260925 10:39:41.999275 29293 sys_catalog.cc:564] T 00000000000000000000000000000000 P 571a89c8ff6d42db92c3bd101bae6c5f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:39:42.001088 29299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 571a89c8ff6d42db92c3bd101bae6c5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "571a89c8ff6d42db92c3bd101bae6c5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "571a89c8ff6d42db92c3bd101bae6c5f" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 43499 } } }
I20260925 10:39:42.001400 29299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 571a89c8ff6d42db92c3bd101bae6c5f [sys.catalog]: This master's current role is: LEADER
I20260925 10:39:42.002156 29300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 571a89c8ff6d42db92c3bd101bae6c5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 571a89c8ff6d42db92c3bd101bae6c5f. Latest consensus state: current_term: 1 leader_uuid: "571a89c8ff6d42db92c3bd101bae6c5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "571a89c8ff6d42db92c3bd101bae6c5f" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 43499 } } }
I20260925 10:39:42.002645 29300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 571a89c8ff6d42db92c3bd101bae6c5f [sys.catalog]: This master's current role is: LEADER
I20260925 10:39:42.007225 29303 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:39:42.011222 29303 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:39:42.020238 29303 catalog_manager.cc:1347] Generated new cluster ID: 8c24d32ccc0544a1b3ead55d3ef15e51
I20260925 10:39:42.020345 29303 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:39:42.026998 29303 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:39:42.028620 29303 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:39:42.054080 29303 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 571a89c8ff6d42db92c3bd101bae6c5f: Generated new TSK 0
I20260925 10:39:42.055013 29303 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 10:39:42.096354 29323 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 10:39:42.096477 29323 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 10:39:42.097692 29296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:42.077924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60716 (local address 127.23.100.62:43499)
0925 10:39:42.078115 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:42.078122 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:42.078533 (+ 411us) server_negotiation.cc:407] Connection header received
0925 10:39:42.078721 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:42.078727 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:42.078812 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:42.078986 (+ 174us) server_negotiation.cc:226] Negotiated authn=JWT
0925 10:39:42.079404 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:42.080547 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:42.081857 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:42.082602 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:42.082725 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:42.083215 (+ 490us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 10:39:42.097026 (+ 13811us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42869/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 10:39:42.097398 (+ 372us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60716: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42869/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":69}
W20260925 10:39:42.097728 29322 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:39:42.077805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:43499 (local address 127.0.0.1:60716)
0925 10:39:42.078427 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:42.078443 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:42.078572 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:42.078987 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:42.078990 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:42.078999 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0925 10:39:42.079252 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:42.079259 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:42.080668 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:42.080672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:42.081724 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:42.081734 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:42.082776 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:42.082780 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:42.082912 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:42.082943 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 10:39:42.097260 (+ 14317us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42869/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 10:39:42.097445 (+ 185us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.100.62:43499: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42869/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":344,"threads_started":1}
W20260925 10:39:42.097824 29296 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60716: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42869/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260925 10:39:42.099198 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 29227
I20260925 10:39:42.100378 29288 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:39:42.193682 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 29227
2026-09-25T10:39:42Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (727 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 10:39:42.248692 23952 mini_oidc.cc:152] Starting JWKS server
I20260925 10:39:42.250561 23952 webserver.cc:460] Webserver started at https://127.0.0.1:39791/ using document root <none> and password file <none>
I20260925 10:39:42.250813 23952 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 10:39:42.251438 23952 webserver.cc:460] Webserver started at http://127.0.0.1:42129/ using document root <none> and password file <none>
2026-09-25T10:39:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:39:42Z Disabled control of system clock
I20260925 10:39:42.278559 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:45311
--webserver_interface=127.23.100.62
--webserver_port=0
--builtin_ntp_servers=127.23.100.20:43773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.62:45311
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39791/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 10:39:42.590821 29330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:39:42.591076 29330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:39:42.591157 29330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:39:42.597930 29330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:39:42.598021 29330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:39:42.598058 29330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:39:42.598105 29330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:39:42.598141 29330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 10:39:42.598201 29330 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 10:39:42.598232 29330 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39791/jwks.json
I20260925 10:39:42.607069 29330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:43773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790332740062128-23952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.62:45311
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:45311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39791/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:39:42.608180 29330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:39:42.609844 29330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:39:42.621013 29336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:42.621083 29335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:42.620976 29338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:42.621915 29330 server_base.cc:1034] running on GCE node
I20260925 10:39:42.622677 29330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:39:42.624145 29330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:39:42.625427 29330 hybrid_clock.cc:648] HybridClock initialized: now 1790332782625381 us; error 54 us; skew 500 ppm
I20260925 10:39:42.625895 29330 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:39:42.628249 29330 webserver.cc:460] Webserver started at http://127.23.100.62:35609/ using document root <none> and password file <none>
I20260925 10:39:42.628983 29330 fs_manager.cc:359] Metadata directory not provided
I20260925 10:39:42.629124 29330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:39:42.629451 29330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:39:42.632242 29330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790332740062128-23952-0/minicluster-data/master-0/data/instance:
uuid: "e64b705124b6478c9baed4ad84552883"
format_stamp: "Formatted at 2026-09-25 10:39:42 on dist-test-slave-dxdd"
server_key: "d58590706d652246dad3de81053fac16"
server_key_iv: "eb4136067a001c62abe809b465c9a33a"
server_key_version: "encryptionkey@0"
I20260925 10:39:42.633021 29330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790332740062128-23952-0/minicluster-data/master-0/wal/instance:
uuid: "e64b705124b6478c9baed4ad84552883"
format_stamp: "Formatted at 2026-09-25 10:39:42 on dist-test-slave-dxdd"
server_key: "d58590706d652246dad3de81053fac16"
server_key_iv: "eb4136067a001c62abe809b465c9a33a"
server_key_version: "encryptionkey@0"
I20260925 10:39:42.637904 29330 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260925 10:39:42.641108 29344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:42.642421 29330 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 10:39:42.642608 29330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790332740062128-23952-0/minicluster-data/master-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790332740062128-23952-0/minicluster-data/master-0/wal
uuid: "e64b705124b6478c9baed4ad84552883"
format_stamp: "Formatted at 2026-09-25 10:39:42 on dist-test-slave-dxdd"
server_key: "d58590706d652246dad3de81053fac16"
server_key_iv: "eb4136067a001c62abe809b465c9a33a"
server_key_version: "encryptionkey@0"
I20260925 10:39:42.642817 29330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790332740062128-23952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790332740062128-23952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790332740062128-23952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:39:42.671124 29330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:39:42.671901 29330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:39:42.672194 29330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:39:42.690860 29330 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.62:45311
I20260925 10:39:42.690881 29395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.62:45311 every 8 connection(s)
I20260925 10:39:42.692104 29330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
I20260925 10:39:42.695174 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 29330
I20260925 10:39:42.695398 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790332740062128-23952-0/minicluster-data/master-0/wal/instance
I20260925 10:39:42.695552 29396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:39:42.696058 23952 external_mini_cluster.cc:1428] Setting key ffafba5a474f086cf0f9f4ab2f15863c
I20260925 10:39:42.710404 29399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:42.697806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58110 (local address 127.23.100.62:45311)
0925 10:39:42.698948 (+ 1142us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:42.698967 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:42.699019 (+ 52us) server_negotiation.cc:407] Connection header received
0925 10:39:42.700652 (+ 1633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:42.700679 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:42.701248 (+ 569us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:42.701694 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:42.702203 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:42.704236 (+ 2033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:42.705481 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:42.706528 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:42.706695 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:42.707711 (+ 1016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:42.707719 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:42.707735 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:39:42.708444 (+ 709us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:42.708533 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:39:42.708556 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:42.708654 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:42.708878 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 10:39:42.709180 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":423,"threads_started":1}
I20260925 10:39:42.710493 29396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e64b705124b6478c9baed4ad84552883: Bootstrap starting.
I20260925 10:39:42.714035 29396 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e64b705124b6478c9baed4ad84552883: Neither blocks nor log segments found. Creating new log.
I20260925 10:39:42.714293 23952 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:39:42.715258 29396 log.cc:824] T 00000000000000000000000000000000 P e64b705124b6478c9baed4ad84552883: Log is configured to *not* fsync() on all Append() calls
I20260925 10:39:42.718147 29396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e64b705124b6478c9baed4ad84552883: No bootstrap required, opened a new log
I20260925 10:39:42.725093 29396 raft_consensus.cc:348] T 00000000000000000000000000000000 P e64b705124b6478c9baed4ad84552883 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e64b705124b6478c9baed4ad84552883" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 45311 } }
I20260925 10:39:42.725407 29396 raft_consensus.cc:374] T 00000000000000000000000000000000 P e64b705124b6478c9baed4ad84552883 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:39:42.725498 29396 raft_consensus.cc:729] T 00000000000000000000000000000000 P e64b705124b6478c9baed4ad84552883 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e64b705124b6478c9baed4ad84552883, State: Initialized, Role: FOLLOWER
I20260925 10:39:42.726111 29396 consensus_queue.cc:260] T 00000000000000000000000000000000 P e64b705124b6478c9baed4ad84552883 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e64b705124b6478c9baed4ad84552883" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 45311 } }
I20260925 10:39:42.726384 29396 raft_consensus.cc:388] T 00000000000000000000000000000000 P e64b705124b6478c9baed4ad84552883 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:39:42.726522 29396 raft_consensus.cc:482] T 00000000000000000000000000000000 P e64b705124b6478c9baed4ad84552883 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:39:42.726699 29396 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e64b705124b6478c9baed4ad84552883 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:39:42.728766 29396 raft_consensus.cc:504] T 00000000000000000000000000000000 P e64b705124b6478c9baed4ad84552883 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e64b705124b6478c9baed4ad84552883" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 45311 } }
I20260925 10:39:42.729278 29396 leader_election.cc:302] T 00000000000000000000000000000000 P e64b705124b6478c9baed4ad84552883 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e64b705124b6478c9baed4ad84552883; no voters:
I20260925 10:39:42.729858 29396 leader_election.cc:288] T 00000000000000000000000000000000 P e64b705124b6478c9baed4ad84552883 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:39:42.729988 29401 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e64b705124b6478c9baed4ad84552883 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:39:42.730734 29401 raft_consensus.cc:686] T 00000000000000000000000000000000 P e64b705124b6478c9baed4ad84552883 [term 1 LEADER]: Becoming Leader. State: Replica: e64b705124b6478c9baed4ad84552883, State: Running, Role: LEADER
I20260925 10:39:42.731217 29401 consensus_queue.cc:237] T 00000000000000000000000000000000 P e64b705124b6478c9baed4ad84552883 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e64b705124b6478c9baed4ad84552883" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 45311 } }
I20260925 10:39:42.731983 29396 sys_catalog.cc:564] T 00000000000000000000000000000000 P e64b705124b6478c9baed4ad84552883 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:39:42.734011 29403 sys_catalog.cc:455] T 00000000000000000000000000000000 P e64b705124b6478c9baed4ad84552883 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e64b705124b6478c9baed4ad84552883. Latest consensus state: current_term: 1 leader_uuid: "e64b705124b6478c9baed4ad84552883" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e64b705124b6478c9baed4ad84552883" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 45311 } } }
I20260925 10:39:42.734549 29403 sys_catalog.cc:458] T 00000000000000000000000000000000 P e64b705124b6478c9baed4ad84552883 [sys.catalog]: This master's current role is: LEADER
I20260925 10:39:42.734020 29402 sys_catalog.cc:455] T 00000000000000000000000000000000 P e64b705124b6478c9baed4ad84552883 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e64b705124b6478c9baed4ad84552883" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e64b705124b6478c9baed4ad84552883" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 45311 } } }
I20260925 10:39:42.738338 29402 sys_catalog.cc:458] T 00000000000000000000000000000000 P e64b705124b6478c9baed4ad84552883 [sys.catalog]: This master's current role is: LEADER
I20260925 10:39:42.739725 29407 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:39:42.745065 29407 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:39:42.753711 29407 catalog_manager.cc:1347] Generated new cluster ID: 94c3b527cc584437b1160bf3f232e23a
I20260925 10:39:42.753844 29407 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:39:42.767854 29407 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:39:42.769052 29407 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:39:42.781054 29407 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e64b705124b6478c9baed4ad84552883: Generated new TSK 0
I20260925 10:39:42.781800 29407 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:39:42.806123 29399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:42.798174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58124 (local address 127.23.100.62:45311)
0925 10:39:42.798393 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:42.798399 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:42.798749 (+ 350us) server_negotiation.cc:407] Connection header received
0925 10:39:42.798939 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:42.798945 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:42.799030 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:42.799199 (+ 169us) server_negotiation.cc:226] Negotiated authn=JWT
0925 10:39:42.799605 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:42.800745 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:42.802022 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:42.802723 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:42.802842 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:42.803258 (+ 416us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 10:39:42.805511 (+ 2253us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39791/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 10:39:42.805836 (+ 325us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58124: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39791/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":68}
W20260925 10:39:42.806238 29399 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58124: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39791/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260925 10:39:42.807862 29425 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:39:42.798034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:45311 (local address 127.0.0.1:58124)
0925 10:39:42.798653 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:42.798668 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:42.798779 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:42.799204 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:42.799207 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:42.799216 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0925 10:39:42.799474 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:42.799481 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:42.800877 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:42.800880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:42.801893 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:42.801903 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:42.802880 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:42.802885 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:42.802997 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:42.803025 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 10:39:42.805696 (+ 2671us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39791/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 10:39:42.807401 (+ 1705us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.100.62:45311: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39791/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":501,"mutex_wait_us":1414,"thread_start_us":349,"threads_started":1}
I20260925 10:39:42.808826 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 29330
I20260925 10:39:42.809877 29391 generic_service.cc:196] Checking for leaks (request via RPC)
E20260925 10:39:42.817965 29426 webserver.cc:533] Webserver: sslize failed:
E20260925 10:39:42.818048 29426 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 10:39:42.900134 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 29330
2026-09-25T10:39:42Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (703 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T10:39:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:39:42Z Disabled control of system clock
I20260925 10:39:42.976778 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:34245
--webserver_interface=127.23.100.62
--webserver_port=0
--builtin_ntp_servers=127.23.100.20:44147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.62:34245
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790332740062128-23952-0/testchainca.cert with env {}
W20260925 10:39:43.289834 29431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:39:43.290143 29431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:39:43.290228 29431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:39:43.297303 29431 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:39:43.297394 29431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:39:43.297428 29431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:39:43.297474 29431 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:39:43.297509 29431 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 10:39:43.297577 29431 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 10:39:43.297616 29431 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 10:39:43.307107 29431 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:44147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790332740062128-23952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.62:34245
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:34245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790332740062128-23952-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:39:43.308220 29431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:39:43.309396 29431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:39:43.321069 29436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:43.321096 29437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:43.321153 29439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:43.322049 29431 server_base.cc:1034] running on GCE node
I20260925 10:39:43.322844 29431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:39:43.324120 29431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:39:43.325347 29431 hybrid_clock.cc:648] HybridClock initialized: now 1790332783325310 us; error 48 us; skew 500 ppm
I20260925 10:39:43.325695 29431 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:39:43.327852 29431 webserver.cc:460] Webserver started at http://127.23.100.62:35661/ using document root <none> and password file <none>
I20260925 10:39:43.328498 29431 fs_manager.cc:359] Metadata directory not provided
I20260925 10:39:43.328624 29431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:39:43.328913 29431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:39:43.331805 29431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790332740062128-23952-0/minicluster-data/master-0/data/instance:
uuid: "a48ab380293b4c1bb7f40bf6008e829f"
format_stamp: "Formatted at 2026-09-25 10:39:43 on dist-test-slave-dxdd"
server_key: "5bed9644c9cd189e5b4370cdcbdd1982"
server_key_iv: "6462169f141944bdb7b675a35efb4666"
server_key_version: "encryptionkey@0"
I20260925 10:39:43.332563 29431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790332740062128-23952-0/minicluster-data/master-0/wal/instance:
uuid: "a48ab380293b4c1bb7f40bf6008e829f"
format_stamp: "Formatted at 2026-09-25 10:39:43 on dist-test-slave-dxdd"
server_key: "5bed9644c9cd189e5b4370cdcbdd1982"
server_key_iv: "6462169f141944bdb7b675a35efb4666"
server_key_version: "encryptionkey@0"
I20260925 10:39:43.337307 29431 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260925 10:39:43.340471 29445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:43.341779 29431 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 10:39:43.341943 29431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790332740062128-23952-0/minicluster-data/master-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790332740062128-23952-0/minicluster-data/master-0/wal
uuid: "a48ab380293b4c1bb7f40bf6008e829f"
format_stamp: "Formatted at 2026-09-25 10:39:43 on dist-test-slave-dxdd"
server_key: "5bed9644c9cd189e5b4370cdcbdd1982"
server_key_iv: "6462169f141944bdb7b675a35efb4666"
server_key_version: "encryptionkey@0"
I20260925 10:39:43.342144 29431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790332740062128-23952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790332740062128-23952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790332740062128-23952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:39:43.367443 29431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:39:43.368264 29431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:39:43.368543 29431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:39:43.386485 29431 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.62:34245
I20260925 10:39:43.386461 29496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.62:34245 every 8 connection(s)
I20260925 10:39:43.387742 29431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
I20260925 10:39:43.391181 29497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:39:43.392686 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 29431
I20260925 10:39:43.392871 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790332740062128-23952-0/minicluster-data/master-0/wal/instance
I20260925 10:39:43.393537 23952 external_mini_cluster.cc:1428] Setting key 71c7bc6ee3e732b471695ae7e1f733a8
I20260925 10:39:43.403985 29497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a48ab380293b4c1bb7f40bf6008e829f: Bootstrap starting.
I20260925 10:39:43.407624 29497 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a48ab380293b4c1bb7f40bf6008e829f: Neither blocks nor log segments found. Creating new log.
I20260925 10:39:43.408308 29500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:43.395460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51540 (local address 127.23.100.62:34245)
0925 10:39:43.396458 (+ 998us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:43.396471 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:43.396524 (+ 53us) server_negotiation.cc:407] Connection header received
0925 10:39:43.398805 (+ 2281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:43.398840 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:43.399381 (+ 541us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:43.399790 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:43.400296 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:43.402368 (+ 2072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:43.403529 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:43.404477 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:43.404638 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:43.405675 (+ 1037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:43.405687 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:43.405703 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:39:43.406388 (+ 685us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:43.406500 (+ 112us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:39:43.406524 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:43.406635 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:43.406796 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 10:39:43.407082 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":281,"threads_started":1}
I20260925 10:39:43.408810 29497 log.cc:824] T 00000000000000000000000000000000 P a48ab380293b4c1bb7f40bf6008e829f: Log is configured to *not* fsync() on all Append() calls
I20260925 10:39:43.411774 29497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a48ab380293b4c1bb7f40bf6008e829f: No bootstrap required, opened a new log
I20260925 10:39:43.411945 23952 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:39:43.419795 29497 raft_consensus.cc:348] T 00000000000000000000000000000000 P a48ab380293b4c1bb7f40bf6008e829f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a48ab380293b4c1bb7f40bf6008e829f" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 34245 } }
I20260925 10:39:43.420135 29497 raft_consensus.cc:374] T 00000000000000000000000000000000 P a48ab380293b4c1bb7f40bf6008e829f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:39:43.420233 29497 raft_consensus.cc:729] T 00000000000000000000000000000000 P a48ab380293b4c1bb7f40bf6008e829f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a48ab380293b4c1bb7f40bf6008e829f, State: Initialized, Role: FOLLOWER
I20260925 10:39:43.420826 29497 consensus_queue.cc:260] T 00000000000000000000000000000000 P a48ab380293b4c1bb7f40bf6008e829f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a48ab380293b4c1bb7f40bf6008e829f" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 34245 } }
I20260925 10:39:43.421049 29497 raft_consensus.cc:388] T 00000000000000000000000000000000 P a48ab380293b4c1bb7f40bf6008e829f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:39:43.421147 29497 raft_consensus.cc:482] T 00000000000000000000000000000000 P a48ab380293b4c1bb7f40bf6008e829f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:39:43.421254 29497 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a48ab380293b4c1bb7f40bf6008e829f [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:39:43.423209 29497 raft_consensus.cc:504] T 00000000000000000000000000000000 P a48ab380293b4c1bb7f40bf6008e829f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a48ab380293b4c1bb7f40bf6008e829f" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 34245 } }
I20260925 10:39:43.423609 29497 leader_election.cc:302] T 00000000000000000000000000000000 P a48ab380293b4c1bb7f40bf6008e829f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a48ab380293b4c1bb7f40bf6008e829f; no voters:
I20260925 10:39:43.424079 29497 leader_election.cc:288] T 00000000000000000000000000000000 P a48ab380293b4c1bb7f40bf6008e829f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:39:43.424229 29502 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a48ab380293b4c1bb7f40bf6008e829f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:39:43.424914 29502 raft_consensus.cc:686] T 00000000000000000000000000000000 P a48ab380293b4c1bb7f40bf6008e829f [term 1 LEADER]: Becoming Leader. State: Replica: a48ab380293b4c1bb7f40bf6008e829f, State: Running, Role: LEADER
I20260925 10:39:43.425478 29502 consensus_queue.cc:237] T 00000000000000000000000000000000 P a48ab380293b4c1bb7f40bf6008e829f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a48ab380293b4c1bb7f40bf6008e829f" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 34245 } }
I20260925 10:39:43.426188 29497 sys_catalog.cc:564] T 00000000000000000000000000000000 P a48ab380293b4c1bb7f40bf6008e829f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:39:43.428434 29504 sys_catalog.cc:455] T 00000000000000000000000000000000 P a48ab380293b4c1bb7f40bf6008e829f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a48ab380293b4c1bb7f40bf6008e829f. Latest consensus state: current_term: 1 leader_uuid: "a48ab380293b4c1bb7f40bf6008e829f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a48ab380293b4c1bb7f40bf6008e829f" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 34245 } } }
I20260925 10:39:43.428814 29504 sys_catalog.cc:458] T 00000000000000000000000000000000 P a48ab380293b4c1bb7f40bf6008e829f [sys.catalog]: This master's current role is: LEADER
I20260925 10:39:43.428534 29503 sys_catalog.cc:455] T 00000000000000000000000000000000 P a48ab380293b4c1bb7f40bf6008e829f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a48ab380293b4c1bb7f40bf6008e829f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a48ab380293b4c1bb7f40bf6008e829f" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 34245 } } }
I20260925 10:39:43.428952 29503 sys_catalog.cc:458] T 00000000000000000000000000000000 P a48ab380293b4c1bb7f40bf6008e829f [sys.catalog]: This master's current role is: LEADER
I20260925 10:39:43.434633 29508 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:39:43.438403 29508 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:39:43.446027 29508 catalog_manager.cc:1347] Generated new cluster ID: 344ea2227ad24e3593899b58835740c9
I20260925 10:39:43.446131 29508 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:39:43.453699 29508 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:39:43.454905 29508 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:39:43.460224 29508 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a48ab380293b4c1bb7f40bf6008e829f: Generated new TSK 0
I20260925 10:39:43.460889 29508 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:39:43.495774 29500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:43.473644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51554 (local address 127.23.100.62:34245)
0925 10:39:43.473852 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:43.473859 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:43.474214 (+ 355us) server_negotiation.cc:407] Connection header received
0925 10:39:43.474403 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:43.474410 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:43.474498 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:43.474638 (+ 140us) server_negotiation.cc:226] Negotiated authn=JWT
0925 10:39:43.475058 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:43.476244 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:43.477614 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:43.478327 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:43.478454 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:43.478868 (+ 414us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 10:39:43.495131 (+ 16263us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 10:39:43.495456 (+ 325us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51554: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":62}
W20260925 10:39:43.495812 29526 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:39:43.473507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:34245 (local address 127.0.0.1:51554)
0925 10:39:43.474113 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:43.474132 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:43.474242 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:43.474663 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:43.474666 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:43.474675 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0925 10:39:43.474920 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:43.474928 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:43.476383 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:43.476386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:43.477488 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:43.477497 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:43.478468 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:43.478472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:43.478579 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:43.478606 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 10:39:43.495349 (+ 16743us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 10:39:43.495532 (+ 183us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.100.62:34245: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":359,"threads_started":1}
W20260925 10:39:43.495898 29500 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51554: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260925 10:39:43.497345 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 29431
I20260925 10:39:43.498476 29492 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:39:43.586684 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 29431
2026-09-25T10:39:43Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (668 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (388 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (1455 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:39:45 dist-test-slave-dxdd krb5kdc[29542](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:39:45 dist-test-slave-dxdd krb5kdc[29542](info): set up 2 sockets
Sep 25 10:39:45 dist-test-slave-dxdd krb5kdc[29542](info): commencing operation
krb5kdc: starting...
W20260925 10:39:46.530934 23952 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.004s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Sep 25 10:39:46 dist-test-slave-dxdd krb5kdc[29542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332786, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:39:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:39:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
I20260925 10:39:46.717700 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:33261
--webserver_interface=127.23.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:33001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.62:33261
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:39:47.023988 29555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:39:47.024247 29555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:39:47.024330 29555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:39:47.031342 29555 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:39:47.031466 29555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:39:47.031572 29555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:39:47.031683 29555 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:39:47.031769 29555 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:39:47.041213 29555 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:33001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.62:33261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:33261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:39:47.042450 29555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:39:47.043713 29555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:39:47.057673 29560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:47.057735 29561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:47.057772 29563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:47.058542 29555 server_base.cc:1034] running on GCE node
I20260925 10:39:47.059273 29555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:39:47.060547 29555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:39:47.061774 29555 hybrid_clock.cc:648] HybridClock initialized: now 1790332787061736 us; error 43 us; skew 500 ppm
I20260925 10:39:47.062103 29555 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:39:47 dist-test-slave-dxdd krb5kdc[29542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332787, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:39:47.070943 29555 init.cc:377] Logged in from keytab as kudu/127.23.100.62@KRBTEST.COM (short username kudu)
I20260925 10:39:47.072341 29555 webserver.cc:460] Webserver started at http://127.23.100.62:45859/ using document root <none> and password file <none>
I20260925 10:39:47.072922 29555 fs_manager.cc:359] Metadata directory not provided
I20260925 10:39:47.073021 29555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:39:47.073283 29555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:39:47.076081 29555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/master-0/data/instance:
uuid: "ba52117d3a824026b6418ef46fb21619"
format_stamp: "Formatted at 2026-09-25 10:39:47 on dist-test-slave-dxdd"
server_key: "b2c13ae621867a8148d2ca00cde45ab1"
server_key_iv: "bf5d31863ac7c722cd15b6e54f76bb72"
server_key_version: "encryptionkey@0"
I20260925 10:39:47.076828 29555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/master-0/wal/instance:
uuid: "ba52117d3a824026b6418ef46fb21619"
format_stamp: "Formatted at 2026-09-25 10:39:47 on dist-test-slave-dxdd"
server_key: "b2c13ae621867a8148d2ca00cde45ab1"
server_key_iv: "bf5d31863ac7c722cd15b6e54f76bb72"
server_key_version: "encryptionkey@0"
I20260925 10:39:47.081562 29555 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 10:39:47.084760 29570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:47.085932 29555 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 10:39:47.086108 29555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/master-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/master-0/wal
uuid: "ba52117d3a824026b6418ef46fb21619"
format_stamp: "Formatted at 2026-09-25 10:39:47 on dist-test-slave-dxdd"
server_key: "b2c13ae621867a8148d2ca00cde45ab1"
server_key_iv: "bf5d31863ac7c722cd15b6e54f76bb72"
server_key_version: "encryptionkey@0"
I20260925 10:39:47.086313 29555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:39:47.116559 29555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:39:47.120271 29555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:39:47.120539 29555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:39:47.138314 29555 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.62:33261
I20260925 10:39:47.138422 29621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.62:33261 every 8 connection(s)
I20260925 10:39:47.139737 29555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
I20260925 10:39:47.143203 29622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:39:47.144245 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 29555
I20260925 10:39:47.144456 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/master-0/wal/instance
I20260925 10:39:47.145237 23952 external_mini_cluster.cc:1428] Setting key 98eb10cc0bac50ab62f8e02ae7ce709b
I20260925 10:39:47.157143 29622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba52117d3a824026b6418ef46fb21619: Bootstrap starting.
Sep 25 10:39:47 dist-test-slave-dxdd krb5kdc[29542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332786, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:39:47.160687 29622 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba52117d3a824026b6418ef46fb21619: Neither blocks nor log segments found. Creating new log.
I20260925 10:39:47.161752 29622 log.cc:824] T 00000000000000000000000000000000 P ba52117d3a824026b6418ef46fb21619: Log is configured to *not* fsync() on all Append() calls
I20260925 10:39:47.164522 29622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba52117d3a824026b6418ef46fb21619: No bootstrap required, opened a new log
I20260925 10:39:47.167552 29625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:47.147247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42026 (local address 127.23.100.62:33261)
0925 10:39:47.147984 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:47.147993 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:47.148030 (+ 37us) server_negotiation.cc:407] Connection header received
0925 10:39:47.149630 (+ 1600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:47.149653 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:47.150006 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:47.150393 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:47.152011 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:47.153273 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:47.155140 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:47.155770 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:47.155896 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:47.160449 (+ 4553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:47.160457 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:47.160470 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:47.160504 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:47.163620 (+ 3116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:47.164353 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:47.164360 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:47.164365 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:47.164540 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:47.164945 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:47.164949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:47.164951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:47.165460 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:47.165754 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:47.166213 (+ 459us) server_negotiation.cc:299] Negotiation successful
0925 10:39:47.166617 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":359,"threads_started":1}
I20260925 10:39:47.172371 29622 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba52117d3a824026b6418ef46fb21619 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba52117d3a824026b6418ef46fb21619" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33261 } }
I20260925 10:39:47.172719 29622 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba52117d3a824026b6418ef46fb21619 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:39:47.172820 29622 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba52117d3a824026b6418ef46fb21619 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba52117d3a824026b6418ef46fb21619, State: Initialized, Role: FOLLOWER
I20260925 10:39:47.173329 29622 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba52117d3a824026b6418ef46fb21619 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba52117d3a824026b6418ef46fb21619" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33261 } }
I20260925 10:39:47.173559 29622 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba52117d3a824026b6418ef46fb21619 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:39:47.173695 29622 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba52117d3a824026b6418ef46fb21619 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:39:47.173851 29622 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba52117d3a824026b6418ef46fb21619 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:39:47.175988 29622 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba52117d3a824026b6418ef46fb21619 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba52117d3a824026b6418ef46fb21619" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33261 } }
I20260925 10:39:47.176453 29622 leader_election.cc:302] T 00000000000000000000000000000000 P ba52117d3a824026b6418ef46fb21619 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba52117d3a824026b6418ef46fb21619; no voters:
I20260925 10:39:47.176971 29622 leader_election.cc:288] T 00000000000000000000000000000000 P ba52117d3a824026b6418ef46fb21619 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:39:47.177104 29627 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba52117d3a824026b6418ef46fb21619 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:39:47.177799 29627 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba52117d3a824026b6418ef46fb21619 [term 1 LEADER]: Becoming Leader. State: Replica: ba52117d3a824026b6418ef46fb21619, State: Running, Role: LEADER
I20260925 10:39:47.178393 29627 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba52117d3a824026b6418ef46fb21619 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba52117d3a824026b6418ef46fb21619" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33261 } }
I20260925 10:39:47.179111 29622 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba52117d3a824026b6418ef46fb21619 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:39:47.181106 29628 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba52117d3a824026b6418ef46fb21619 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba52117d3a824026b6418ef46fb21619" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba52117d3a824026b6418ef46fb21619" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33261 } } }
I20260925 10:39:47.181424 29628 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba52117d3a824026b6418ef46fb21619 [sys.catalog]: This master's current role is: LEADER
I20260925 10:39:47.182984 29633 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:39:47.182978 29629 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba52117d3a824026b6418ef46fb21619 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba52117d3a824026b6418ef46fb21619. Latest consensus state: current_term: 1 leader_uuid: "ba52117d3a824026b6418ef46fb21619" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba52117d3a824026b6418ef46fb21619" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33261 } } }
I20260925 10:39:47.183279 29629 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba52117d3a824026b6418ef46fb21619 [sys.catalog]: This master's current role is: LEADER
I20260925 10:39:47.187705 29633 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:39:47.196125 29633 catalog_manager.cc:1347] Generated new cluster ID: 78177dbaa96f4339a04b48ab034e19d5
I20260925 10:39:47.196280 29633 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:39:47.220865 29633 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:39:47.222065 29633 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:39:47.232626 29633 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba52117d3a824026b6418ef46fb21619: Generated new TSK 0
I20260925 10:39:47.233371 29633 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab.
Entry for principal kudu/127.23.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab.
I20260925 10:39:47.291559 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.1:0
--local_ip_for_outbound_sockets=127.23.100.1
--webserver_interface=127.23.100.1
--webserver_port=0
--tserver_master_addrs=127.23.100.62:33261
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab
--principal=kudu/127.23.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:33001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:39:47.621583 29648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:39:47.621860 29648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:39:47.622042 29648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:39:47.629184 29648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:39:47.629278 29648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:39:47.629381 29648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.1
I20260925 10:39:47.638734 29648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:33001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab
--principal=kudu/127.23.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.1
--webserver_port=0
--tserver_master_addrs=127.23.100.62:33261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.1
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:39:47.640018 29648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:39:47.641301 29648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:39:47.654161 29653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:47.654106 29654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:47.654198 29656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:47.654973 29648 server_base.cc:1034] running on GCE node
I20260925 10:39:47.655515 29648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:39:47.656355 29648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:39:47.657544 29648 hybrid_clock.cc:648] HybridClock initialized: now 1790332787657503 us; error 36 us; skew 500 ppm
I20260925 10:39:47.657824 29648 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:39:47 dist-test-slave-dxdd krb5kdc[29542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332787, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:39:47.660851 29648 init.cc:377] Logged in from keytab as kudu/127.23.100.1@KRBTEST.COM (short username kudu)
I20260925 10:39:47.662117 29648 webserver.cc:460] Webserver started at http://127.23.100.1:35187/ using document root <none> and password file <none>
I20260925 10:39:47.662789 29648 fs_manager.cc:359] Metadata directory not provided
I20260925 10:39:47.662889 29648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:39:47.663133 29648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:39:47.665783 29648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/ts-0/data/instance:
uuid: "1021523fe007425394f4ebc36fc7f4cf"
format_stamp: "Formatted at 2026-09-25 10:39:47 on dist-test-slave-dxdd"
server_key: "37941eda4ab9c11aacdb0b08441ba854"
server_key_iv: "c1f5943590f69fa64662feb07dd5b112"
server_key_version: "encryptionkey@0"
I20260925 10:39:47.666627 29648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/ts-0/wal/instance:
uuid: "1021523fe007425394f4ebc36fc7f4cf"
format_stamp: "Formatted at 2026-09-25 10:39:47 on dist-test-slave-dxdd"
server_key: "37941eda4ab9c11aacdb0b08441ba854"
server_key_iv: "c1f5943590f69fa64662feb07dd5b112"
server_key_version: "encryptionkey@0"
I20260925 10:39:47.671610 29648 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260925 10:39:47.674777 29663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:47.675963 29648 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.002s
I20260925 10:39:47.676086 29648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/ts-0/wal
uuid: "1021523fe007425394f4ebc36fc7f4cf"
format_stamp: "Formatted at 2026-09-25 10:39:47 on dist-test-slave-dxdd"
server_key: "37941eda4ab9c11aacdb0b08441ba854"
server_key_iv: "c1f5943590f69fa64662feb07dd5b112"
server_key_version: "encryptionkey@0"
I20260925 10:39:47.676255 29648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:39:47.687852 29648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:39:47.692056 29648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:39:47.692276 29648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:39:47.693068 29648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:39:47.694191 29648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:39:47.694259 29648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:47.694394 29648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:39:47.694444 29648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:47.725541 29648 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.1:37531
I20260925 10:39:47.725586 29775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.1:37531 every 8 connection(s)
I20260925 10:39:47.726770 29648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
I20260925 10:39:47.731984 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 29648
I20260925 10:39:47.732208 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/ts-0/wal/instance
I20260925 10:39:47.732969 23952 external_mini_cluster.cc:1428] Setting key 1dbe34f06093eb3086f121226e31827e
Sep 25 10:39:47 dist-test-slave-dxdd krb5kdc[29542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332787, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.1@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:39:47.747793 29778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:47.729149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:43983 (local address 127.23.100.62:33261)
0925 10:39:47.729563 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:47.729569 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:47.730577 (+ 1008us) server_negotiation.cc:407] Connection header received
0925 10:39:47.732468 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:47.732473 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:47.732543 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:47.732628 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:47.734409 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:47.735365 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:47.737835 (+ 2470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:47.738233 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:47.738405 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:47.742285 (+ 3880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:47.742290 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:47.742293 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:47.742369 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:47.744850 (+ 2481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:47.745534 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:47.745543 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:47.745546 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:47.745656 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:47.746067 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:47.746072 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:47.746075 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:47.746398 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:47.746555 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:47.747225 (+ 670us) server_negotiation.cc:299] Negotiation successful
0925 10:39:47.747436 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":303,"thread_start_us":200,"threads_started":1}
I20260925 10:39:47.748694 29779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:47.729697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:33261 (local address 127.23.100.1:43983)
0925 10:39:47.730422 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:47.730460 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:47.732054 (+ 1594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:47.732930 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:47.732944 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:47.733652 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:47.734240 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:47.734262 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:47.735512 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:47.735519 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:47.737609 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:47.737629 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:47.738468 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:47.738477 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:47.738795 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:47.739501 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:47.739533 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:47.742077 (+ 2544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:47.744986 (+ 2909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:47.744995 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:47.745011 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:47.745411 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:47.745758 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:47.745764 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:47.745768 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:47.745949 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:47.746525 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:47.746533 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:47.746849 (+ 316us) client_negotiation.cc:769] Sending connection context
0925 10:39:47.747101 (+ 252us) client_negotiation.cc:240] Negotiation successful
0925 10:39:47.747593 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":268,"threads_started":1}
I20260925 10:39:47.750268 29776 heartbeater.cc:347] Connected to a master server at 127.23.100.62:33261
I20260925 10:39:47.750685 29776 heartbeater.cc:464] Registering TS with master...
I20260925 10:39:47.751549 29776 heartbeater.cc:511] Master 127.23.100.62:33261 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.100.2@KRBTEST.COM; defaulting to no policy
I20260925 10:39:47.754029 29587 ts_manager.cc:194] Registered new tserver with Master: 1021523fe007425394f4ebc36fc7f4cf (127.23.100.1:37531)
I20260925 10:39:47.755518 29587 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.1@KRBTEST.COM'} at 127.23.100.1:43983
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab.
Entry for principal kudu/127.23.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab.
I20260925 10:39:47.779649 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.2:0
--local_ip_for_outbound_sockets=127.23.100.2
--webserver_interface=127.23.100.2
--webserver_port=0
--tserver_master_addrs=127.23.100.62:33261
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab
--principal=kudu/127.23.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:33001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:39:48.096688 29782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:39:48.096935 29782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:39:48.097025 29782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:39:48.103971 29782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:39:48.104066 29782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:39:48.104184 29782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.2
I20260925 10:39:48.113312 29782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:33001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab
--principal=kudu/127.23.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.2
--webserver_port=0
--tserver_master_addrs=127.23.100.62:33261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.2
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:39:48.114502 29782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:39:48.115787 29782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:39:48.128237 29790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:48.128284 29787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:48.128252 29788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:48.129055 29782 server_base.cc:1034] running on GCE node
I20260925 10:39:48.129680 29782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:39:48.130679 29782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:39:48.131909 29782 hybrid_clock.cc:648] HybridClock initialized: now 1790332788131871 us; error 52 us; skew 500 ppm
I20260925 10:39:48.132225 29782 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:39:48 dist-test-slave-dxdd krb5kdc[29542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332788, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:39:48.135262 29782 init.cc:377] Logged in from keytab as kudu/127.23.100.2@KRBTEST.COM (short username kudu)
I20260925 10:39:48.136615 29782 webserver.cc:460] Webserver started at http://127.23.100.2:44467/ using document root <none> and password file <none>
I20260925 10:39:48.137226 29782 fs_manager.cc:359] Metadata directory not provided
I20260925 10:39:48.137320 29782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:39:48.137562 29782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:39:48.140514 29782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/ts-1/data/instance:
uuid: "6324c84d631e4bb0a86402e49be48118"
format_stamp: "Formatted at 2026-09-25 10:39:48 on dist-test-slave-dxdd"
server_key: "6c8e4a7c308120989d483082c40b786b"
server_key_iv: "8d69ae0e4e2b497003ca72b5b138cbce"
server_key_version: "encryptionkey@0"
I20260925 10:39:48.141325 29782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/ts-1/wal/instance:
uuid: "6324c84d631e4bb0a86402e49be48118"
format_stamp: "Formatted at 2026-09-25 10:39:48 on dist-test-slave-dxdd"
server_key: "6c8e4a7c308120989d483082c40b786b"
server_key_iv: "8d69ae0e4e2b497003ca72b5b138cbce"
server_key_version: "encryptionkey@0"
I20260925 10:39:48.146351 29782 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260925 10:39:48.149725 29797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:48.151129 29782 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:39:48.151293 29782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/ts-1/wal
uuid: "6324c84d631e4bb0a86402e49be48118"
format_stamp: "Formatted at 2026-09-25 10:39:48 on dist-test-slave-dxdd"
server_key: "6c8e4a7c308120989d483082c40b786b"
server_key_iv: "8d69ae0e4e2b497003ca72b5b138cbce"
server_key_version: "encryptionkey@0"
I20260925 10:39:48.151484 29782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:39:48.171792 29782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:39:48.175661 29782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:39:48.175999 29782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:39:48.176931 29782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:39:48.178200 29782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:39:48.178305 29782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:48.178484 29782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:39:48.178556 29782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:48.211905 29782 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.2:39937
I20260925 10:39:48.211992 29909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.2:39937 every 8 connection(s)
I20260925 10:39:48.213197 29782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
I20260925 10:39:48.216935 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 29782
I20260925 10:39:48.217130 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/ts-1/wal/instance
I20260925 10:39:48.217597 23952 external_mini_cluster.cc:1428] Setting key 46a460561aab0ab2b7621aa8ee215241
Sep 25 10:39:48 dist-test-slave-dxdd krb5kdc[29542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332788, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.2@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:39:48.233556 29778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:48.215743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:53553 (local address 127.23.100.62:33261)
0925 10:39:48.215962 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:48.215968 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:48.217119 (+ 1151us) server_negotiation.cc:407] Connection header received
0925 10:39:48.219225 (+ 2106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:48.219228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:48.219297 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:48.219419 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:48.220919 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:48.221623 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:48.223180 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:48.223597 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:48.223730 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:48.227640 (+ 3910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:48.227645 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:48.227648 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:48.227679 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:48.230380 (+ 2701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:48.231245 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:48.231248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:48.231251 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:48.231375 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:48.231855 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:48.231857 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:48.231859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:48.232089 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:48.232210 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:48.233066 (+ 856us) server_negotiation.cc:299] Negotiation successful
0925 10:39:48.233300 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 10:39:48.235095 29912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:48.216214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:33261 (local address 127.23.100.2:53553)
0925 10:39:48.216908 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:48.216944 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:48.218932 (+ 1988us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:48.219706 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:48.219719 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:48.220231 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:48.220760 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:48.220776 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:48.221765 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:48.221770 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:48.223037 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:48.223048 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:48.223812 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:48.223818 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:48.224045 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:48.224703 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:48.224727 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:48.227428 (+ 2701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:48.230499 (+ 3071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:48.230510 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:48.230527 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:48.231123 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:48.231500 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:48.231508 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:48.231515 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:48.231748 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:48.232212 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:48.232223 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:48.232671 (+ 448us) client_negotiation.cc:769] Sending connection context
0925 10:39:48.232962 (+ 291us) client_negotiation.cc:240] Negotiation successful
0925 10:39:48.233512 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":293,"threads_started":1}
I20260925 10:39:48.236549 29910 heartbeater.cc:347] Connected to a master server at 127.23.100.62:33261
I20260925 10:39:48.236841 29910 heartbeater.cc:464] Registering TS with master...
I20260925 10:39:48.237498 29910 heartbeater.cc:511] Master 127.23.100.62:33261 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.100.3@KRBTEST.COM; defaulting to no policy
I20260925 10:39:48.239082 29587 ts_manager.cc:194] Registered new tserver with Master: 6324c84d631e4bb0a86402e49be48118 (127.23.100.2:39937)
I20260925 10:39:48.239856 29587 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.2@KRBTEST.COM'} at 127.23.100.2:53553
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab.
Entry for principal kudu/127.23.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab.
I20260925 10:39:48.262949 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.3:0
--local_ip_for_outbound_sockets=127.23.100.3
--webserver_interface=127.23.100.3
--webserver_port=0
--tserver_master_addrs=127.23.100.62:33261
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab
--principal=kudu/127.23.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:33001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:39:48.576004 29915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:39:48.576298 29915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:39:48.576480 29915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:39:48.583781 29915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:39:48.583935 29915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:39:48.584123 29915 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.3
I20260925 10:39:48.593972 29915 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:33001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab
--principal=kudu/127.23.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.3
--webserver_port=0
--tserver_master_addrs=127.23.100.62:33261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.3
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:39:48.595260 29915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:39:48.596570 29915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:39:48.609546 29920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:48.609488 29921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:48.609567 29923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:48.610329 29915 server_base.cc:1034] running on GCE node
I20260925 10:39:48.610917 29915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:39:48.611721 29915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:39:48.612919 29915 hybrid_clock.cc:648] HybridClock initialized: now 1790332788612897 us; error 38 us; skew 500 ppm
I20260925 10:39:48.613245 29915 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:39:48 dist-test-slave-dxdd krb5kdc[29542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332788, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:39:48.616192 29915 init.cc:377] Logged in from keytab as kudu/127.23.100.3@KRBTEST.COM (short username kudu)
I20260925 10:39:48.617470 29915 webserver.cc:460] Webserver started at http://127.23.100.3:45641/ using document root <none> and password file <none>
I20260925 10:39:48.618139 29915 fs_manager.cc:359] Metadata directory not provided
I20260925 10:39:48.618281 29915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:39:48.618598 29915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:39:48.621330 29915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/ts-2/data/instance:
uuid: "71d0d7c1bd164aa4b551615c0c6d4ee6"
format_stamp: "Formatted at 2026-09-25 10:39:48 on dist-test-slave-dxdd"
server_key: "55637e8cdabdef011fa2e9fb5ef728ba"
server_key_iv: "53c767cee46101414a9e91053d4fe194"
server_key_version: "encryptionkey@0"
I20260925 10:39:48.622087 29915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/ts-2/wal/instance:
uuid: "71d0d7c1bd164aa4b551615c0c6d4ee6"
format_stamp: "Formatted at 2026-09-25 10:39:48 on dist-test-slave-dxdd"
server_key: "55637e8cdabdef011fa2e9fb5ef728ba"
server_key_iv: "53c767cee46101414a9e91053d4fe194"
server_key_version: "encryptionkey@0"
I20260925 10:39:48.626878 29915 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 10:39:48.629933 29930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:48.631088 29915 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 10:39:48.631264 29915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/ts-2/wal
uuid: "71d0d7c1bd164aa4b551615c0c6d4ee6"
format_stamp: "Formatted at 2026-09-25 10:39:48 on dist-test-slave-dxdd"
server_key: "55637e8cdabdef011fa2e9fb5ef728ba"
server_key_iv: "53c767cee46101414a9e91053d4fe194"
server_key_version: "encryptionkey@0"
I20260925 10:39:48.631453 29915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:39:48.657270 29915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:39:48.661104 29915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:39:48.661391 29915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:39:48.662286 29915 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:39:48.663614 29915 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:39:48.663712 29915 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:48.663852 29915 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:39:48.663941 29915 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:48.696076 29915 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.3:41395
I20260925 10:39:48.696113 30042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.3:41395 every 8 connection(s)
I20260925 10:39:48.697293 29915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
I20260925 10:39:48.700980 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 29915
I20260925 10:39:48.701160 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/minicluster-data/ts-2/wal/instance
I20260925 10:39:48.701644 23952 external_mini_cluster.cc:1428] Setting key 7f4954a6f097c52b3588c3d174dd0290
Sep 25 10:39:48 dist-test-slave-dxdd krb5kdc[29542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332788, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.3@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:39:48.715950 29778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:48.699699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:39103 (local address 127.23.100.62:33261)
0925 10:39:48.699900 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:48.699906 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:48.701064 (+ 1158us) server_negotiation.cc:407] Connection header received
0925 10:39:48.702969 (+ 1905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:48.702975 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:48.703079 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:48.703186 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:48.704595 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:48.705323 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:48.706916 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:48.707284 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:48.707414 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:48.710786 (+ 3372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:48.710791 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:48.710794 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:48.710825 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:48.713186 (+ 2361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:48.713832 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:48.713842 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:48.713844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:48.713951 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:48.714368 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:48.714373 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:48.714377 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:48.714617 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:48.714796 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:48.715414 (+ 618us) server_negotiation.cc:299] Negotiation successful
0925 10:39:48.715631 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 10:39:48.717008 30045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:48.700237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:33261 (local address 127.23.100.3:39103)
0925 10:39:48.700920 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:48.700955 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:48.702721 (+ 1766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:48.703432 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:48.703452 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:48.703927 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:48.704443 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:48.704458 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:48.705511 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:48.705518 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:48.706773 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:48.706788 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:48.707489 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:48.707498 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:48.707721 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:48.708360 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:48.708380 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:48.710585 (+ 2205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:48.713312 (+ 2727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:48.713320 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:48.713334 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:48.713712 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:48.714056 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:48.714061 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:48.714065 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:48.714235 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:48.714762 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:48.714769 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:48.715059 (+ 290us) client_negotiation.cc:769] Sending connection context
0925 10:39:48.715299 (+ 240us) client_negotiation.cc:240] Negotiation successful
0925 10:39:48.715744 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":288,"threads_started":1}
I20260925 10:39:48.718194 30043 heartbeater.cc:347] Connected to a master server at 127.23.100.62:33261
I20260925 10:39:48.718478 30043 heartbeater.cc:464] Registering TS with master...
I20260925 10:39:48.719123 30043 heartbeater.cc:511] Master 127.23.100.62:33261 requested a full tablet report, sending...
I20260925 10:39:48.720362 29587 ts_manager.cc:194] Registered new tserver with Master: 71d0d7c1bd164aa4b551615c0c6d4ee6 (127.23.100.3:41395)
I20260925 10:39:48.721149 29587 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.3@KRBTEST.COM'} at 127.23.100.3:39103
I20260925 10:39:48.728870 23952 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/krb5kdc/test-admin.keytab.
Sep 25 10:39:48 dist-test-slave-dxdd krb5kdc[29542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:39:48.756078 23952 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
I20260925 10:39:48.758445 29776 heartbeater.cc:503] Master 127.23.100.62:33261 was elected leader, sending a full tablet report...
W20260925 10:39:48.761317 30050 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:39:48.758588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49580 (local address 127.23.100.1:37531)
0925 10:39:48.759369 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:48.759381 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:48.759412 (+ 31us) server_negotiation.cc:407] Connection header received
0925 10:39:48.759472 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:48.759480 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:48.759760 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:48.759932 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:48.760897 (+ 965us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49580: BlockingRecv error: recv got EOF from 127.0.0.1:49580 (error 108)
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":399,"threads_started":1}
W20260925 10:39:48.761389 23952 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/krb5kdc/krb5cc' of type FILE
Sep 25 10:39:48 dist-test-slave-dxdd krb5kdc[29542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:39:48.762972 23952 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 10:39:48 dist-test-slave-dxdd krb5kdc[29542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:39:48.764386 23952 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 10:39:48 dist-test-slave-dxdd krb5kdc[29542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.1@KRBTEST.COM
I20260925 10:39:48.779151 30050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:48.766012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49588 (local address 127.23.100.1:37531)
0925 10:39:48.766192 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:48.766198 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:48.766396 (+ 198us) server_negotiation.cc:407] Connection header received
0925 10:39:48.766598 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:48.766602 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:48.766658 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:48.766765 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:48.767765 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:48.768691 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:48.769878 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:48.770458 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:48.770582 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:48.773905 (+ 3323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:48.773920 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:48.773934 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:48.773970 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:48.776409 (+ 2439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:48.777037 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:48.777045 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:48.777047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:48.777155 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:48.777514 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:48.777536 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:48.777541 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:48.777866 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:48.778169 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:48.778526 (+ 357us) server_negotiation.cc:299] Negotiation successful
0925 10:39:48.778799 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260925 10:39:48.784593 30050 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:39:48.783135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49594 (local address 127.23.100.1:37531)
0925 10:39:48.783311 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:48.783317 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:48.783424 (+ 107us) server_negotiation.cc:407] Connection header received
0925 10:39:48.783538 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:48.783541 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:48.783598 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:48.783669 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:48.784373 (+ 704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49594: BlockingRecv error: recv got EOF from 127.0.0.1:49594 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260925 10:39:48.784725 23952 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/krb5kdc/krb5cc' of type FILE
Sep 25 10:39:48 dist-test-slave-dxdd krb5kdc[29542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:39:48.786033 23952 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 10:39:48 dist-test-slave-dxdd krb5kdc[29542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:39:48.787307 23952 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 10:39:48 dist-test-slave-dxdd krb5kdc[29542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.1@KRBTEST.COM
I20260925 10:39:48.799845 30050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:48.788489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49610 (local address 127.23.100.1:37531)
0925 10:39:48.788657 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:48.788665 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:48.788899 (+ 234us) server_negotiation.cc:407] Connection header received
0925 10:39:48.789062 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:48.789066 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:48.789123 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:48.789219 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:48.790128 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:48.790869 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:48.792028 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:48.792441 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:48.792568 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:48.795658 (+ 3090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:48.795662 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:48.795665 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:48.795699 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:48.797831 (+ 2132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:48.798428 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:48.798431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:48.798433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:48.798537 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:48.798871 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:48.798874 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:48.798876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:48.799123 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:48.799305 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:48.799504 (+ 199us) server_negotiation.cc:299] Negotiation successful
0925 10:39:48.799639 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260925 10:39:48.803756 30050 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:39:48.802106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49618 (local address 127.23.100.1:37531)
0925 10:39:48.802289 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:48.802297 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:48.802557 (+ 260us) server_negotiation.cc:407] Connection header received
0925 10:39:48.802733 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:48.802737 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:48.802797 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:48.802892 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:48.803503 (+ 611us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49618: BlockingRecv error: recv got EOF from 127.0.0.1:49618 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260925 10:39:48.803921 23952 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790332740062128-23952-0/krb5kdc/krb5cc' of type FILE
Sep 25 10:39:48 dist-test-slave-dxdd krb5kdc[29542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:39:48.805258 23952 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 10:39:48 dist-test-slave-dxdd krb5kdc[29542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:39:48.806519 23952 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 10:39:48 dist-test-slave-dxdd krb5kdc[29542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.1@KRBTEST.COM
I20260925 10:39:48.819415 30050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:48.807593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49620 (local address 127.23.100.1:37531)
0925 10:39:48.807799 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:48.807805 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:48.808004 (+ 199us) server_negotiation.cc:407] Connection header received
0925 10:39:48.808102 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:48.808105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:48.808153 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:48.808226 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:48.809068 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:48.809785 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:48.810981 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:48.811405 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:48.811495 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:48.814599 (+ 3104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:48.814603 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:48.814607 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:48.814633 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:48.817116 (+ 2483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:48.817793 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:48.817797 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:48.817799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:48.817938 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:48.818243 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:48.818245 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:48.818247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:48.818557 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:48.818795 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:48.818932 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 10:39:48.819121 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 10:39:48.820803 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 29648
I20260925 10:39:48.831204 30050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:48.821418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49626 (local address 127.23.100.1:37531)
0925 10:39:48.821546 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:48.821550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:48.821908 (+ 358us) server_negotiation.cc:407] Connection header received
0925 10:39:48.822015 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:48.822019 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:48.822063 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:48.822133 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:48.822950 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:48.823558 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:48.824717 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:48.825108 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:48.825200 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:48.826980 (+ 1780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:48.826985 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:48.826988 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:48.827013 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:48.829285 (+ 2272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:48.829766 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:48.829769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:48.829771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:48.829876 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:48.830146 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:48.830149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:48.830151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:48.830416 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:48.830579 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:48.830832 (+ 253us) server_negotiation.cc:299] Negotiation successful
0925 10:39:48.830960 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 10:39:48.831740 29771 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:39:48.914088 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 29648
I20260925 10:39:48.929811 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 29782
Sep 25 10:39:48 dist-test-slave-dxdd krb5kdc[29542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.2@KRBTEST.COM
I20260925 10:39:48.944618 30063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:48.930653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51556 (local address 127.23.100.2:39937)
0925 10:39:48.931214 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:48.931222 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:48.931247 (+ 25us) server_negotiation.cc:407] Connection header received
0925 10:39:48.931309 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:48.931315 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:48.931518 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:48.931643 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:48.932527 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:48.933519 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:48.934876 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:48.935452 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:48.935542 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:48.939014 (+ 3472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:48.939025 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:48.939037 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:48.939083 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:48.942137 (+ 3054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:48.942932 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:48.942938 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:48.942940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:48.943064 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:48.943385 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:48.943388 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:48.943390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:48.943709 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:48.943943 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:48.944154 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 10:39:48.944338 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":298,"threads_started":1}
I20260925 10:39:48.945930 29905 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:39:49.027320 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 29782
I20260925 10:39:49.043371 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 29915
Sep 25 10:39:49 dist-test-slave-dxdd krb5kdc[29542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.3@KRBTEST.COM
I20260925 10:39:49.060333 30065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:49.044400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53972 (local address 127.23.100.3:41395)
0925 10:39:49.044992 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:49.044999 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:49.045026 (+ 27us) server_negotiation.cc:407] Connection header received
0925 10:39:49.045109 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:49.045120 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:49.045450 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:49.045628 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:49.047179 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:49.048138 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:49.049580 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:49.050214 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:49.050373 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:49.055360 (+ 4987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:49.055370 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:49.055384 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:49.055415 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:49.057883 (+ 2468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:49.058545 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:49.058558 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:49.058560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:49.058675 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:49.058984 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:49.058986 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:49.058988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:49.059336 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:49.059563 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:49.059738 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 10:39:49.059988 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":305,"threads_started":1}
I20260925 10:39:49.061522 30038 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:39:49.145071 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 29915
I20260925 10:39:49.163105 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 29555
I20260925 10:39:49.164258 29617 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:39:49.265395 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 29555
2026-09-25T10:39:49Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3841 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:39:49 dist-test-slave-dxdd krb5kdc[30070](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:39:49 dist-test-slave-dxdd krb5kdc[30070](info): set up 2 sockets
Sep 25 10:39:49 dist-test-slave-dxdd krb5kdc[30070](info): commencing operation
krb5kdc: starting...
W20260925 10:39:50.338927 23952 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Sep 25 10:39:50 dist-test-slave-dxdd krb5kdc[30070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:39:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:39:50Z Disabled control of system clock
WARNING: no policy specified for oryx/127.23.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/oryx_127.23.100.62.keytab.
Entry for principal oryx/127.23.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/oryx_127.23.100.62.keytab.
I20260925 10:39:50.527796 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:33367
--webserver_interface=127.23.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/oryx_127.23.100.62.keytab
--principal=oryx/127.23.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:37749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.62:33367
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:39:50.832338 30083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:39:50.832594 30083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:39:50.832746 30083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:39:50.839538 30083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:39:50.839627 30083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:39:50.839669 30083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:39:50.839725 30083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:39:50.839766 30083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:39:50.849156 30083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:37749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.62:33367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/oryx_127.23.100.62.keytab
--principal=oryx/127.23.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:33367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:39:50.850296 30083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:39:50.851553 30083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:39:50.863121 30088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:50.863145 30091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:50.863078 30089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:50.863798 30083 server_base.cc:1034] running on GCE node
I20260925 10:39:50.864495 30083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:39:50.865675 30083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:39:50.866863 30083 hybrid_clock.cc:648] HybridClock initialized: now 1790332790866825 us; error 35 us; skew 500 ppm
I20260925 10:39:50.867189 30083 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:39:50 dist-test-slave-dxdd krb5kdc[30070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332790, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:39:50.870088 30083 init.cc:377] Logged in from keytab as oryx/127.23.100.62@KRBTEST.COM (short username oryx)
I20260925 10:39:50.871512 30083 webserver.cc:460] Webserver started at http://127.23.100.62:35297/ using document root <none> and password file <none>
I20260925 10:39:50.872133 30083 fs_manager.cc:359] Metadata directory not provided
I20260925 10:39:50.872268 30083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:39:50.872562 30083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:39:50.875401 30083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-0/data/instance:
uuid: "a0d0cc835da447eda23faee26d106480"
format_stamp: "Formatted at 2026-09-25 10:39:50 on dist-test-slave-dxdd"
server_key: "1e75b773e5f8e44ab8240c3bd24e2e59"
server_key_iv: "b9e69e483e30034df8ceb9f8c074d6ab"
server_key_version: "encryptionkey@0"
I20260925 10:39:50.876119 30083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-0/wal/instance:
uuid: "a0d0cc835da447eda23faee26d106480"
format_stamp: "Formatted at 2026-09-25 10:39:50 on dist-test-slave-dxdd"
server_key: "1e75b773e5f8e44ab8240c3bd24e2e59"
server_key_iv: "b9e69e483e30034df8ceb9f8c074d6ab"
server_key_version: "encryptionkey@0"
I20260925 10:39:50.880976 30083 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 10:39:50.884220 30098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:50.885425 30083 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 10:39:50.885594 30083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-0/wal
uuid: "a0d0cc835da447eda23faee26d106480"
format_stamp: "Formatted at 2026-09-25 10:39:50 on dist-test-slave-dxdd"
server_key: "1e75b773e5f8e44ab8240c3bd24e2e59"
server_key_iv: "b9e69e483e30034df8ceb9f8c074d6ab"
server_key_version: "encryptionkey@0"
I20260925 10:39:50.885778 30083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:39:50.910149 30083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:39:50.913892 30083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:39:50.914175 30083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:39:50.931847 30083 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.62:33367
I20260925 10:39:50.931888 30149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.62:33367 every 8 connection(s)
I20260925 10:39:50.933141 30083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
I20260925 10:39:50.934509 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 30083
I20260925 10:39:50.934703 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-0/wal/instance
I20260925 10:39:50.935535 23952 external_mini_cluster.cc:1428] Setting key 345f9d59cfd2ce60920e2611f8640473
I20260925 10:39:50.937641 30150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 10:39:50 dist-test-slave-dxdd krb5kdc[30070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM
I20260925 10:39:50.950379 30150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480: Bootstrap starting.
I20260925 10:39:50.954207 30150 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480: Neither blocks nor log segments found. Creating new log.
I20260925 10:39:50.955503 30150 log.cc:824] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480: Log is configured to *not* fsync() on all Append() calls
I20260925 10:39:50.958141 30153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:50.937435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38310 (local address 127.23.100.62:33367)
0925 10:39:50.938141 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:50.938152 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:50.938187 (+ 35us) server_negotiation.cc:407] Connection header received
0925 10:39:50.939778 (+ 1591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:50.939797 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:50.940143 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:50.940475 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:50.942130 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:50.943539 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:50.945245 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:50.945854 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:50.945978 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:50.950290 (+ 4312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:50.950332 (+ 42us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:50.950349 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:50.950398 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:50.953947 (+ 3549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:50.954642 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:50.954652 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:50.954664 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:50.954848 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:50.955242 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:50.955248 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:50.955252 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:50.955902 (+ 650us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:50.956270 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:50.956602 (+ 332us) server_negotiation.cc:299] Negotiation successful
0925 10:39:50.956943 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":328,"threads_started":1}
I20260925 10:39:50.958786 30150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480: No bootstrap required, opened a new log
I20260925 10:39:50.966171 30150 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0d0cc835da447eda23faee26d106480" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33367 } }
I20260925 10:39:50.966578 30150 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:39:50.966651 30150 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d0cc835da447eda23faee26d106480, State: Initialized, Role: FOLLOWER
I20260925 10:39:50.967275 30150 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0d0cc835da447eda23faee26d106480" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33367 } }
I20260925 10:39:50.967592 30150 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:39:50.967720 30150 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:39:50.967921 30150 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:39:50.970153 30150 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0d0cc835da447eda23faee26d106480" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33367 } }
I20260925 10:39:50.970628 30150 leader_election.cc:302] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0d0cc835da447eda23faee26d106480; no voters:
I20260925 10:39:50.971155 30150 leader_election.cc:288] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:39:50.971284 30155 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:39:50.971966 30155 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480 [term 1 LEADER]: Becoming Leader. State: Replica: a0d0cc835da447eda23faee26d106480, State: Running, Role: LEADER
I20260925 10:39:50.972541 30155 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0d0cc835da447eda23faee26d106480" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33367 } }
I20260925 10:39:50.973095 30150 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:39:50.975026 30156 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0d0cc835da447eda23faee26d106480" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0d0cc835da447eda23faee26d106480" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33367 } } }
I20260925 10:39:50.975338 30156 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480 [sys.catalog]: This master's current role is: LEADER
I20260925 10:39:50.976272 30162 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:39:50.976060 30157 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0d0cc835da447eda23faee26d106480. Latest consensus state: current_term: 1 leader_uuid: "a0d0cc835da447eda23faee26d106480" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0d0cc835da447eda23faee26d106480" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33367 } } }
I20260925 10:39:50.977458 30157 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480 [sys.catalog]: This master's current role is: LEADER
I20260925 10:39:50.983188 30162 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:39:50.991921 30162 catalog_manager.cc:1347] Generated new cluster ID: 20e688bc766742f58dcb363b0a466e9f
I20260925 10:39:50.992024 30162 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:39:51.002830 30162 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:39:51.004526 30162 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:39:51.022464 30162 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480: Generated new TSK 0
I20260925 10:39:51.023451 30162 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.23.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/oryx_127.23.100.1.keytab.
Entry for principal oryx/127.23.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/oryx_127.23.100.1.keytab.
I20260925 10:39:51.075686 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.1:0
--local_ip_for_outbound_sockets=127.23.100.1
--webserver_interface=127.23.100.1
--webserver_port=0
--tserver_master_addrs=127.23.100.62:33367
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/oryx_127.23.100.1.keytab
--principal=oryx/127.23.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:37749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:39:51.387176 30176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:39:51.387477 30176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:39:51.387643 30176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:39:51.394903 30176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:39:51.395036 30176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:39:51.395210 30176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.1
I20260925 10:39:51.404666 30176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:37749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/oryx_127.23.100.1.keytab
--principal=oryx/127.23.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.1
--webserver_port=0
--tserver_master_addrs=127.23.100.62:33367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.1
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:39:51.405932 30176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:39:51.407262 30176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:39:51.419672 30182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:51.419684 30181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:51.419926 30184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:51.420353 30176 server_base.cc:1034] running on GCE node
I20260925 10:39:51.420976 30176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:39:51.421823 30176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:39:51.423036 30176 hybrid_clock.cc:648] HybridClock initialized: now 1790332791422995 us; error 57 us; skew 500 ppm
I20260925 10:39:51.423322 30176 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:39:51 dist-test-slave-dxdd krb5kdc[30070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332791, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:39:51.426220 30176 init.cc:377] Logged in from keytab as oryx/127.23.100.1@KRBTEST.COM (short username oryx)
I20260925 10:39:51.427574 30176 webserver.cc:460] Webserver started at http://127.23.100.1:44391/ using document root <none> and password file <none>
I20260925 10:39:51.428244 30176 fs_manager.cc:359] Metadata directory not provided
I20260925 10:39:51.428390 30176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:39:51.428676 30176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:39:51.431464 30176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-0/data/instance:
uuid: "e23f60bcefdf41b1843f6d5f5fcce1fb"
format_stamp: "Formatted at 2026-09-25 10:39:51 on dist-test-slave-dxdd"
server_key: "117395e44da70038a09c0d0aaa45d569"
server_key_iv: "010d9c3612da0708346bd8ad9e6471d0"
server_key_version: "encryptionkey@0"
I20260925 10:39:51.432178 30176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-0/wal/instance:
uuid: "e23f60bcefdf41b1843f6d5f5fcce1fb"
format_stamp: "Formatted at 2026-09-25 10:39:51 on dist-test-slave-dxdd"
server_key: "117395e44da70038a09c0d0aaa45d569"
server_key_iv: "010d9c3612da0708346bd8ad9e6471d0"
server_key_version: "encryptionkey@0"
I20260925 10:39:51.437287 30176 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 10:39:51.440531 30191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:51.441641 30176 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 10:39:51.441807 30176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-0/wal
uuid: "e23f60bcefdf41b1843f6d5f5fcce1fb"
format_stamp: "Formatted at 2026-09-25 10:39:51 on dist-test-slave-dxdd"
server_key: "117395e44da70038a09c0d0aaa45d569"
server_key_iv: "010d9c3612da0708346bd8ad9e6471d0"
server_key_version: "encryptionkey@0"
I20260925 10:39:51.442005 30176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:39:51.461086 30176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:39:51.464704 30176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:39:51.464979 30176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:39:51.465844 30176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:39:51.467125 30176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:39:51.467226 30176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:51.467370 30176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:39:51.467439 30176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:51.499646 30176 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.1:43887
I20260925 10:39:51.499725 30303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.1:43887 every 8 connection(s)
I20260925 10:39:51.500808 30176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
I20260925 10:39:51.505020 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 30176
I20260925 10:39:51.505226 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-0/wal/instance
I20260925 10:39:51.505833 23952 external_mini_cluster.cc:1428] Setting key 3b59bfce678d2a128ab62720806fff43
Sep 25 10:39:51 dist-test-slave-dxdd krb5kdc[30070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332791, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.1@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM
I20260925 10:39:51.520467 30306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:51.503249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:57707 (local address 127.23.100.62:33367)
0925 10:39:51.503679 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:51.503683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:51.504678 (+ 995us) server_negotiation.cc:407] Connection header received
0925 10:39:51.506737 (+ 2059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:51.506741 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:51.506805 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:51.506925 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:51.508483 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:51.509252 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:51.510792 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:51.511178 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:51.511311 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:51.514915 (+ 3604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:51.514920 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:51.514923 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:51.514953 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:51.517450 (+ 2497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:51.518337 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:51.518344 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:51.518351 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:51.518465 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:51.518907 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:51.518910 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:51.518911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:51.519146 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:51.519261 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:51.520067 (+ 806us) server_negotiation.cc:299] Negotiation successful
0925 10:39:51.520216 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":219,"threads_started":1}
I20260925 10:39:51.522064 30307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:51.503822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:33367 (local address 127.23.100.1:57707)
0925 10:39:51.504513 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:51.504550 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:51.506459 (+ 1909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:51.507170 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:51.507177 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:51.507801 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:51.508316 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:51.508332 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:51.509391 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:51.509398 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:51.510648 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:51.510659 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:51.511391 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:51.511401 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:51.511644 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:51.512298 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:51.512322 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:51.514711 (+ 2389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:51.517579 (+ 2868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:51.517590 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:51.517606 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:51.518198 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:51.518563 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:51.518570 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:51.518575 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:51.518800 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:51.519297 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:51.519307 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:51.519720 (+ 413us) client_negotiation.cc:769] Sending connection context
0925 10:39:51.519992 (+ 272us) client_negotiation.cc:240] Negotiation successful
0925 10:39:51.520548 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":268,"threads_started":1}
I20260925 10:39:51.523733 30304 heartbeater.cc:347] Connected to a master server at 127.23.100.62:33367
I20260925 10:39:51.524037 30304 heartbeater.cc:464] Registering TS with master...
I20260925 10:39:51.524863 30304 heartbeater.cc:511] Master 127.23.100.62:33367 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.23.100.2@KRBTEST.COM; defaulting to no policy
I20260925 10:39:51.527031 30115 ts_manager.cc:194] Registered new tserver with Master: e23f60bcefdf41b1843f6d5f5fcce1fb (127.23.100.1:43887)
I20260925 10:39:51.528334 30115 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.100.1@KRBTEST.COM'} at 127.23.100.1:57707
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.100.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/oryx_127.23.100.2.keytab.
Entry for principal oryx/127.23.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/oryx_127.23.100.2.keytab.
I20260925 10:39:51.550763 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.2:0
--local_ip_for_outbound_sockets=127.23.100.2
--webserver_interface=127.23.100.2
--webserver_port=0
--tserver_master_addrs=127.23.100.62:33367
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/oryx_127.23.100.2.keytab
--principal=oryx/127.23.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:37749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:39:51.857857 30310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:39:51.858116 30310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:39:51.858206 30310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:39:51.864923 30310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:39:51.865020 30310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:39:51.865127 30310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.2
I20260925 10:39:51.874608 30310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:37749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/oryx_127.23.100.2.keytab
--principal=oryx/127.23.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.2
--webserver_port=0
--tserver_master_addrs=127.23.100.62:33367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.2
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:39:51.875785 30310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:39:51.877079 30310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:39:51.889756 30316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:51.889712 30315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:51.890038 30318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:51.890743 30310 server_base.cc:1034] running on GCE node
I20260925 10:39:51.891325 30310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:39:51.892201 30310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:39:51.893451 30310 hybrid_clock.cc:648] HybridClock initialized: now 1790332791893396 us; error 47 us; skew 500 ppm
I20260925 10:39:51.893771 30310 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:39:51 dist-test-slave-dxdd krb5kdc[30070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332791, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:39:51.896852 30310 init.cc:377] Logged in from keytab as oryx/127.23.100.2@KRBTEST.COM (short username oryx)
I20260925 10:39:51.898257 30310 webserver.cc:460] Webserver started at http://127.23.100.2:41649/ using document root <none> and password file <none>
I20260925 10:39:51.898937 30310 fs_manager.cc:359] Metadata directory not provided
I20260925 10:39:51.899078 30310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:39:51.899377 30310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:39:51.902153 30310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-1/data/instance:
uuid: "6bb6c395684244c384deb64401dafeba"
format_stamp: "Formatted at 2026-09-25 10:39:51 on dist-test-slave-dxdd"
server_key: "8188abee403cdffef829628b3a0fc882"
server_key_iv: "1540d4d6b998a0e4d076b314a6c94028"
server_key_version: "encryptionkey@0"
I20260925 10:39:51.903000 30310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-1/wal/instance:
uuid: "6bb6c395684244c384deb64401dafeba"
format_stamp: "Formatted at 2026-09-25 10:39:51 on dist-test-slave-dxdd"
server_key: "8188abee403cdffef829628b3a0fc882"
server_key_iv: "1540d4d6b998a0e4d076b314a6c94028"
server_key_version: "encryptionkey@0"
I20260925 10:39:51.908047 30310 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 10:39:51.911315 30325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:51.912616 30310 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 10:39:51.912807 30310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-1/wal
uuid: "6bb6c395684244c384deb64401dafeba"
format_stamp: "Formatted at 2026-09-25 10:39:51 on dist-test-slave-dxdd"
server_key: "8188abee403cdffef829628b3a0fc882"
server_key_iv: "1540d4d6b998a0e4d076b314a6c94028"
server_key_version: "encryptionkey@0"
I20260925 10:39:51.913005 30310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:39:51.941882 30310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:39:51.946265 30310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:39:51.946614 30310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:39:51.947618 30310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:39:51.948956 30310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:39:51.949059 30310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:51.949209 30310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:39:51.949296 30310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:51.982836 30310 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.2:37033
I20260925 10:39:51.982888 30437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.2:37033 every 8 connection(s)
I20260925 10:39:51.984123 30310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
I20260925 10:39:51.988001 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 30310
I20260925 10:39:51.988180 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-1/wal/instance
I20260925 10:39:51.988756 23952 external_mini_cluster.cc:1428] Setting key aba281c46a16f5d4d20348a11025e2a8
Sep 25 10:39:51 dist-test-slave-dxdd krb5kdc[30070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332791, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.2@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM
I20260925 10:39:52.005715 30306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:51.986655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:44867 (local address 127.23.100.62:33367)
0925 10:39:51.986889 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:51.986896 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:51.988308 (+ 1412us) server_negotiation.cc:407] Connection header received
0925 10:39:51.990649 (+ 2341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:51.990653 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:51.990741 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:51.990876 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:51.992374 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:51.993538 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:51.995120 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:51.995852 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:51.995988 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:51.999845 (+ 3857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:51.999849 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:51.999853 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:51.999883 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:52.002563 (+ 2680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:52.003440 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:52.003444 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:52.003446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:52.003575 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:52.004043 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:52.004046 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:52.004048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:52.004289 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:52.004405 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:52.005239 (+ 834us) server_negotiation.cc:299] Negotiation successful
0925 10:39:52.005399 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 10:39:52.007238 30440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:51.987300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:33367 (local address 127.23.100.2:44867)
0925 10:39:51.988139 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:51.988198 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:51.990391 (+ 2193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:51.991108 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:51.991115 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:51.991640 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:51.992194 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:51.992211 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:51.993680 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:51.993684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:51.994964 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:51.994980 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:51.996056 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:51.996060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:51.996249 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:51.996935 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:51.996958 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:51.999640 (+ 2682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:52.002687 (+ 3047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:52.002698 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:52.002717 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:52.003322 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:52.003682 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:52.003689 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:52.003696 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:52.003925 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:52.004412 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:52.004422 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:52.004847 (+ 425us) client_negotiation.cc:769] Sending connection context
0925 10:39:52.005148 (+ 301us) client_negotiation.cc:240] Negotiation successful
0925 10:39:52.005719 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":324,"threads_started":1}
I20260925 10:39:52.008785 30438 heartbeater.cc:347] Connected to a master server at 127.23.100.62:33367
I20260925 10:39:52.009140 30438 heartbeater.cc:464] Registering TS with master...
I20260925 10:39:52.010049 30438 heartbeater.cc:511] Master 127.23.100.62:33367 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.23.100.3@KRBTEST.COM; defaulting to no policy
I20260925 10:39:52.011598 30115 ts_manager.cc:194] Registered new tserver with Master: 6bb6c395684244c384deb64401dafeba (127.23.100.2:37033)
I20260925 10:39:52.012391 30115 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.100.2@KRBTEST.COM'} at 127.23.100.2:44867
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.100.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/oryx_127.23.100.3.keytab.
Entry for principal oryx/127.23.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/oryx_127.23.100.3.keytab.
I20260925 10:39:52.036988 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.3:0
--local_ip_for_outbound_sockets=127.23.100.3
--webserver_interface=127.23.100.3
--webserver_port=0
--tserver_master_addrs=127.23.100.62:33367
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/oryx_127.23.100.3.keytab
--principal=oryx/127.23.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:37749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:39:52.346611 30443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:39:52.346892 30443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:39:52.347057 30443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:39:52.353895 30443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:39:52.354029 30443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:39:52.354220 30443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.3
I20260925 10:39:52.363132 30443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:37749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/oryx_127.23.100.3.keytab
--principal=oryx/127.23.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.3
--webserver_port=0
--tserver_master_addrs=127.23.100.62:33367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.3
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:39:52.364297 30443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:39:52.365520 30443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:39:52.377825 30451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:52.377940 30448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:52.377907 30449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:52.378923 30443 server_base.cc:1034] running on GCE node
I20260925 10:39:52.379459 30443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:39:52.380249 30443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:39:52.381443 30443 hybrid_clock.cc:648] HybridClock initialized: now 1790332792381403 us; error 30 us; skew 500 ppm
I20260925 10:39:52.381707 30443 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:39:52 dist-test-slave-dxdd krb5kdc[30070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332792, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:39:52.389951 30443 init.cc:377] Logged in from keytab as oryx/127.23.100.3@KRBTEST.COM (short username oryx)
I20260925 10:39:52.391296 30443 webserver.cc:460] Webserver started at http://127.23.100.3:46419/ using document root <none> and password file <none>
I20260925 10:39:52.391953 30443 fs_manager.cc:359] Metadata directory not provided
I20260925 10:39:52.392079 30443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:39:52.392395 30443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:39:52.395236 30443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-2/data/instance:
uuid: "2971592598aa4a4b8f73d4450d8c92cc"
format_stamp: "Formatted at 2026-09-25 10:39:52 on dist-test-slave-dxdd"
server_key: "389fde49a89293496c10784e78fc951c"
server_key_iv: "6f622726b512d53bf61132f5228422cd"
server_key_version: "encryptionkey@0"
I20260925 10:39:52.395991 30443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-2/wal/instance:
uuid: "2971592598aa4a4b8f73d4450d8c92cc"
format_stamp: "Formatted at 2026-09-25 10:39:52 on dist-test-slave-dxdd"
server_key: "389fde49a89293496c10784e78fc951c"
server_key_iv: "6f622726b512d53bf61132f5228422cd"
server_key_version: "encryptionkey@0"
I20260925 10:39:52.400727 30443 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 10:39:52.404063 30458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:52.405282 30443 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 10:39:52.405447 30443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-2/wal
uuid: "2971592598aa4a4b8f73d4450d8c92cc"
format_stamp: "Formatted at 2026-09-25 10:39:52 on dist-test-slave-dxdd"
server_key: "389fde49a89293496c10784e78fc951c"
server_key_iv: "6f622726b512d53bf61132f5228422cd"
server_key_version: "encryptionkey@0"
I20260925 10:39:52.405611 30443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:39:52.439920 30443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:39:52.444026 30443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:39:52.444300 30443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:39:52.445215 30443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:39:52.446647 30443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:39:52.446744 30443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:52.446903 30443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:39:52.446961 30443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:52.478732 30443 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.3:35055
I20260925 10:39:52.478793 30570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.3:35055 every 8 connection(s)
I20260925 10:39:52.479984 30443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
I20260925 10:39:52.486089 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 30443
I20260925 10:39:52.486245 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-2/wal/instance
I20260925 10:39:52.486697 23952 external_mini_cluster.cc:1428] Setting key 12b5f46382b8b963463a526452d6bf36
Sep 25 10:39:52 dist-test-slave-dxdd krb5kdc[30070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332792, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.3@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM
I20260925 10:39:52.498627 30306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:52.482416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:37645 (local address 127.23.100.62:33367)
0925 10:39:52.482595 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:52.482599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:52.483718 (+ 1119us) server_negotiation.cc:407] Connection header received
0925 10:39:52.485443 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:52.485449 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:52.485536 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:52.485697 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:52.487329 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:52.488467 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:52.490086 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:52.490613 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:52.490709 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:52.493828 (+ 3119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:52.493832 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:52.493836 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:52.493865 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:52.496234 (+ 2369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:52.496839 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:52.496842 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:52.496844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:52.496949 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:52.497284 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:52.497287 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:52.497289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:52.497514 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:52.497618 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:52.498174 (+ 556us) server_negotiation.cc:299] Negotiation successful
0925 10:39:52.498391 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 10:39:52.499600 30573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:52.482906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:33367 (local address 127.23.100.3:37645)
0925 10:39:52.483565 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:52.483605 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:52.485185 (+ 1580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:52.485911 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:52.485919 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:52.486530 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:52.487117 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:52.487159 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:52.488615 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:52.488619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:52.489969 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:52.489980 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:52.490828 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:52.490833 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:52.491019 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:52.491658 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:52.491684 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:52.493662 (+ 1978us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:52.496350 (+ 2688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:52.496355 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:52.496365 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:52.496748 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:52.497043 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:52.497046 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:52.497048 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:52.497200 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:52.497612 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:52.497617 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:52.497883 (+ 266us) client_negotiation.cc:769] Sending connection context
0925 10:39:52.498086 (+ 203us) client_negotiation.cc:240] Negotiation successful
0925 10:39:52.498501 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":262,"threads_started":1}
I20260925 10:39:52.500995 30571 heartbeater.cc:347] Connected to a master server at 127.23.100.62:33367
I20260925 10:39:52.501283 30571 heartbeater.cc:464] Registering TS with master...
I20260925 10:39:52.502097 30571 heartbeater.cc:511] Master 127.23.100.62:33367 requested a full tablet report, sending...
I20260925 10:39:52.503568 30114 ts_manager.cc:194] Registered new tserver with Master: 2971592598aa4a4b8f73d4450d8c92cc (127.23.100.3:35055)
I20260925 10:39:52.504401 30114 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.100.3@KRBTEST.COM'} at 127.23.100.3:37645
I20260925 10:39:52.513692 23952 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:39:52.531004 30304 heartbeater.cc:503] Master 127.23.100.62:33367 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.23.100.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.100.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.100.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/oryx_127.23.100.61.keytab.
Entry for principal oryx/127.23.100.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/oryx_127.23.100.61.keytab.
I20260925 10:39:52.559787 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.61:38343
--webserver_interface=127.23.100.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/oryx_127.23.100.61.keytab
--principal=oryx/127.23.100.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:37749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.62:33367,127.23.100.61:38343
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:39:52.863898 30576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:39:52.864296 30576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:39:52.864507 30576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:39:52.871647 30576 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:39:52.871779 30576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:39:52.871873 30576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:39:52.871966 30576 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:39:52.872058 30576 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:39:52.881641 30576 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:37749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.62:33367,127.23.100.61:38343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/oryx_127.23.100.61.keytab
--principal=oryx/127.23.100.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.61:38343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.100.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
W20260925 10:39:52.881961 30576 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.23.100.62:33367,127.23.100.61:38343'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 10:39:52.883121 30576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:39:52.884330 30576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:39:52.895638 30584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:52.895704 30581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:52.895747 30582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:52.896456 30576 server_base.cc:1034] running on GCE node
I20260925 10:39:52.897133 30576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:39:52.898380 30576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:39:52.899624 30576 hybrid_clock.cc:648] HybridClock initialized: now 1790332792899551 us; error 58 us; skew 500 ppm
I20260925 10:39:52.899930 30576 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:39:52 dist-test-slave-dxdd krb5kdc[30070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332792, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:39:52.902819 30576 init.cc:377] Logged in from keytab as oryx/127.23.100.61@KRBTEST.COM (short username oryx)
I20260925 10:39:52.904187 30576 webserver.cc:460] Webserver started at http://127.23.100.61:44881/ using document root <none> and password file <none>
I20260925 10:39:52.904754 30576 fs_manager.cc:359] Metadata directory not provided
I20260925 10:39:52.904832 30576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:39:52.905093 30576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:39:52.907850 30576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-1/data/instance:
uuid: "d1dd4804029c47089cbf66ab6106c8c5"
format_stamp: "Formatted at 2026-09-25 10:39:52 on dist-test-slave-dxdd"
server_key: "80862cd31aa8377b400a8d20a41f786a"
server_key_iv: "4a5a2dcd8b09d9522819e2d3e12c4a5f"
server_key_version: "encryptionkey@0"
I20260925 10:39:52.908519 30576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-1/wal/instance:
uuid: "d1dd4804029c47089cbf66ab6106c8c5"
format_stamp: "Formatted at 2026-09-25 10:39:52 on dist-test-slave-dxdd"
server_key: "80862cd31aa8377b400a8d20a41f786a"
server_key_iv: "4a5a2dcd8b09d9522819e2d3e12c4a5f"
server_key_version: "encryptionkey@0"
I20260925 10:39:52.912997 30576 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 10:39:52.916093 30591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:52.917506 30576 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:39:52.917686 30576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-1/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-1/wal
uuid: "d1dd4804029c47089cbf66ab6106c8c5"
format_stamp: "Formatted at 2026-09-25 10:39:52 on dist-test-slave-dxdd"
server_key: "80862cd31aa8377b400a8d20a41f786a"
server_key_iv: "4a5a2dcd8b09d9522819e2d3e12c4a5f"
server_key_version: "encryptionkey@0"
I20260925 10:39:52.917909 30576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:39:52.934708 30576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:39:52.938584 30576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:39:52.938823 30576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:39:52.956565 30576 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.61:38343
I20260925 10:39:52.956595 30642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.61:38343 every 8 connection(s)
I20260925 10:39:52.957823 30576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-1/data/info.pb
I20260925 10:39:52.961216 30643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:39:52.965992 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 30576
I20260925 10:39:52.967995 30643 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33367 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:39:52 dist-test-slave-dxdd krb5kdc[30070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332792, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.61@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM
Sep 25 10:39:52 dist-test-slave-dxdd krb5kdc[30070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.100.61@KRBTEST.COM
I20260925 10:39:52.988345 30306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:52.970740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42190 (local address 127.23.100.62:33367)
0925 10:39:52.970918 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:52.970924 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:52.971413 (+ 489us) server_negotiation.cc:407] Connection header received
0925 10:39:52.971666 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:52.971672 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:52.971748 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:52.971870 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:52.973087 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:52.974266 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:52.975918 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:52.976349 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:52.976486 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:52.980825 (+ 4339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:52.980830 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:52.980835 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:52.980872 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:52.984458 (+ 3586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:52.985395 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:52.985401 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:52.985405 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:52.985549 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:52.986291 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:52.986296 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:52.986300 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:52.986765 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:52.986924 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:52.987770 (+ 846us) server_negotiation.cc:299] Negotiation successful
0925 10:39:52.987998 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 10:39:52.989317 30647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:52.970607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:33367 (local address 127.0.0.1:42190)
0925 10:39:52.971260 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:52.971298 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:52.971449 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:52.971927 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:52.971933 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:52.972510 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:52.972873 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:52.972898 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:52.974448 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:52.974462 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:52.975773 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:52.975784 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:52.976598 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:52.976605 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:52.976878 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:52.977586 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:52.977606 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:52.980623 (+ 3017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:52.984621 (+ 3998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:52.984633 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:52.984645 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:52.985266 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:52.985691 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:52.985697 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:52.985700 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:52.985932 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:52.986924 (+ 992us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:52.986934 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:52.987290 (+ 356us) client_negotiation.cc:769] Sending connection context
0925 10:39:52.987645 (+ 355us) client_negotiation.cc:240] Negotiation successful
0925 10:39:52.988007 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"mutex_wait_us":26,"thread_start_us":279,"threads_started":1}
I20260925 10:39:52.989758 30646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:52.968186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34384 (local address 127.23.100.61:38343)
0925 10:39:52.968915 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:52.968926 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:52.968963 (+ 37us) server_negotiation.cc:407] Connection header received
0925 10:39:52.970773 (+ 1810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:52.970798 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:52.971235 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:52.971555 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:52.973036 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:52.974942 (+ 1906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:52.976313 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:52.977003 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:52.977173 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:52.982569 (+ 5396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:52.982583 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:52.982600 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:52.982640 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:52.986396 (+ 3756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:52.987099 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:52.987108 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:52.987111 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:52.987260 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:52.987701 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:52.987706 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:52.987709 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:52.988092 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:52.988410 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:52.988737 (+ 327us) server_negotiation.cc:299] Negotiation successful
0925 10:39:52.988967 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":545,"thread_start_us":341,"threads_started":1}
I20260925 10:39:52.991559 30643 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.61" port: 38343 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:39:52 dist-test-slave-dxdd krb5kdc[30070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332792, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.61@KRBTEST.COM for oryx/127.23.100.61@KRBTEST.COM
I20260925 10:39:53.003336 30647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:52.992230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.61:38343 (local address 127.0.0.1:34392)
0925 10:39:52.992430 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:52.992449 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:52.992552 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:52.992860 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:52.992863 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:52.993140 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:52.993260 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:52.993267 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:52.994200 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:52.994203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:52.995097 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:52.995107 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:52.995736 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:52.995740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:52.995886 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:52.996339 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:52.996353 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:52.998196 (+ 1843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:53.001117 (+ 2921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:53.001121 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:53.001123 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:53.001443 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:53.001976 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:53.001980 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:53.001982 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:53.002123 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:53.002699 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:53.002702 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:53.002826 (+ 124us) client_negotiation.cc:769] Sending connection context
0925 10:39:53.002933 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 10:39:53.003116 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 10:39:53.003690 30646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:52.992452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34392 (local address 127.23.100.61:38343)
0925 10:39:52.992608 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:52.992614 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:52.992632 (+ 18us) server_negotiation.cc:407] Connection header received
0925 10:39:52.992679 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:52.992684 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:52.992762 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:52.992841 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:52.993390 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:52.994074 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:52.995212 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:52.995611 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:52.995716 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:52.998454 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:52.998458 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:52.998461 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:52.998489 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:53.001004 (+ 2515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:53.001560 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:53.001564 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:53.001566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:53.001706 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:53.002246 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:53.002249 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:53.002251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:53.002595 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:53.002702 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:53.003085 (+ 383us) server_negotiation.cc:299] Negotiation successful
0925 10:39:53.003253 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 10:39:53.008386 30643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5: Bootstrap starting.
I20260925 10:39:53.011413 30643 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5: Neither blocks nor log segments found. Creating new log.
I20260925 10:39:53.012379 30643 log.cc:824] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5: Log is configured to *not* fsync() on all Append() calls
I20260925 10:39:53.014918 30438 heartbeater.cc:503] Master 127.23.100.62:33367 was elected leader, sending a full tablet report...
I20260925 10:39:53.015149 30643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5: No bootstrap required, opened a new log
I20260925 10:39:53.022769 30643 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d0cc835da447eda23faee26d106480" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33367 } } peers { permanent_uuid: "d1dd4804029c47089cbf66ab6106c8c5" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 38343 } }
I20260925 10:39:53.023061 30643 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:39:53.023164 30643 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1dd4804029c47089cbf66ab6106c8c5, State: Initialized, Role: FOLLOWER
I20260925 10:39:53.023715 30643 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d0cc835da447eda23faee26d106480" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33367 } } peers { permanent_uuid: "d1dd4804029c47089cbf66ab6106c8c5" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 38343 } }
I20260925 10:39:53.024498 30649 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d0cc835da447eda23faee26d106480" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33367 } } peers { permanent_uuid: "d1dd4804029c47089cbf66ab6106c8c5" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 38343 } } }
I20260925 10:39:53.024860 30649 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:39:53.025471 30643 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 10:39:53.034480 30660 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:39:53.034615 30660 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:39:53.035967 30576 master_runner.cc:428] Detected that this master d1dd4804029c47089cbf66ab6106c8c5 is joining an existing cluster
I20260925 10:39:53.036062 30576 master_runner.cc:432] Initiating AddMaster RPC to add 127.23.100.61:38343
I20260925 10:39:53.056805 30306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:53.039660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42198 (local address 127.23.100.62:33367)
0925 10:39:53.039871 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:53.039881 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:53.040504 (+ 623us) server_negotiation.cc:407] Connection header received
0925 10:39:53.040677 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:53.040683 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:53.040767 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:53.040881 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:53.042515 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:53.043711 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:53.045822 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:53.046521 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:53.046625 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:53.049290 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:53.049296 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:53.049300 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:53.049336 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:53.053822 (+ 4486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:53.054653 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:53.054658 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:53.054662 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:53.054818 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:53.055253 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:53.055258 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:53.055261 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:53.055597 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:53.055725 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:53.056245 (+ 520us) server_negotiation.cc:299] Negotiation successful
0925 10:39:53.056479 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 10:39:53.056792 30646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:53.040644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34394 (local address 127.23.100.61:38343)
0925 10:39:53.040797 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:53.040802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:53.041461 (+ 659us) server_negotiation.cc:407] Connection header received
0925 10:39:53.041627 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:53.041631 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:53.041692 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:53.041785 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:53.042964 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:53.043984 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:53.045983 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:53.046641 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:53.046720 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:53.049277 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:53.049283 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:53.049287 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:53.049317 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:53.053562 (+ 4245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:53.054522 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:53.054528 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:53.054531 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:53.054671 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:53.055089 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:53.055094 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:53.055097 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:53.055487 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:53.055654 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:53.056185 (+ 531us) server_negotiation.cc:299] Negotiation successful
0925 10:39:53.056398 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":34,"spinlock_wait_cycles":896}
I20260925 10:39:53.057971 30667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:53.040789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.61:38343 (local address 127.0.0.1:34394)
0925 10:39:53.041358 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:53.041376 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:53.041515 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:53.041798 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:53.041802 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:53.042475 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:53.042808 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:53.042821 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:53.044205 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:53.044212 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:53.045800 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:53.045811 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:53.047038 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:53.047044 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:53.047190 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:53.047912 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:53.047935 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:53.049119 (+ 1184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:53.053706 (+ 4587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:53.053728 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:53.053734 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:53.054385 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:53.054802 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:53.054806 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:53.054809 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:53.054978 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:53.055757 (+ 779us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:53.055762 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:53.055910 (+ 148us) client_negotiation.cc:769] Sending connection context
0925 10:39:53.057046 (+ 1136us) client_negotiation.cc:240] Negotiation successful
0925 10:39:53.057399 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"mutex_wait_us":5,"spinlock_wait_cycles":31488,"thread_start_us":287,"threads_started":1}
I20260925 10:39:53.058564 30666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:53.039723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:33367 (local address 127.0.0.1:42198)
0925 10:39:53.040393 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:53.040418 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:53.040551 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:53.041065 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:53.041071 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:53.041816 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:53.042342 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:53.042356 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:53.043871 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:53.043876 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:53.045659 (+ 1783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:53.045678 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:53.046899 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:53.046908 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:53.047094 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:53.047762 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:53.047787 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:53.049072 (+ 1285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:53.053961 (+ 4889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:53.053964 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:53.053968 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:53.054523 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:53.054970 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:53.054974 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:53.054977 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:53.055123 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:53.055800 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:53.055804 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:53.055954 (+ 150us) client_negotiation.cc:769] Sending connection context
0925 10:39:53.057046 (+ 1092us) client_negotiation.cc:240] Negotiation successful
0925 10:39:53.057568 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"mutex_wait_us":229,"thread_start_us":360,"threads_started":1}
WARNING: no policy specified for oryx/127.23.100.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.100.60@KRBTEST.COM" created.
Sep 25 10:39:53 dist-test-slave-dxdd krb5kdc[30070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332790, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.62@KRBTEST.COM for oryx/127.23.100.61@KRBTEST.COM
I20260925 10:39:53.084800 30669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:53.065305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.61:38343 (local address 127.0.0.1:34402)
0925 10:39:53.066903 (+ 1598us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:53.066947 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:53.067166 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:53.067696 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:53.067706 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:53.068668 (+ 962us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:53.068968 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:53.068984 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:53.070421 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:53.070428 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:53.072146 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:53.072164 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:53.073226 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:53.073231 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:53.073412 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:53.074235 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:53.074268 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:53.077484 (+ 3216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:53.081552 (+ 4068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:53.081566 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:53.081573 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:53.082121 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:53.082636 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:53.082641 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:53.082645 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:53.082851 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:53.083552 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:53.083561 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:53.083780 (+ 219us) client_negotiation.cc:769] Sending connection context
0925 10:39:53.083994 (+ 214us) client_negotiation.cc:240] Negotiation successful
0925 10:39:53.084330 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1320,"thread_start_us":1166,"threads_started":1}
I20260925 10:39:53.084851 30646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:53.065459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34402 (local address 127.23.100.61:38343)
0925 10:39:53.065659 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:53.065669 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:53.067030 (+ 1361us) server_negotiation.cc:407] Connection header received
0925 10:39:53.067385 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:53.067391 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:53.067508 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:53.067619 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:53.069150 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:53.070255 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:53.072311 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:53.073064 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:53.073190 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:53.077667 (+ 4477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:53.077674 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:53.077679 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:53.077715 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:53.081379 (+ 3664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:53.082264 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:53.082270 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:53.082276 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:53.082501 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:53.082979 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:53.082987 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:53.082994 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:53.083383 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:53.083537 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:53.084111 (+ 574us) server_negotiation.cc:299] Negotiation successful
0925 10:39:53.084384 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 10:39:53.087517 30114 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.23.100.61:38343: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "d1dd4804029c47089cbf66ab6106c8c5"
member_type: NON_VOTER
last_known_addr {
host: "127.23.100.61"
port: 38343
}
attrs {
promote: true
}
}
dest_uuid: "a0d0cc835da447eda23faee26d106480"
cas_config_opid_index: -1
I20260925 10:39:53.088738 30114 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a0d0cc835da447eda23faee26d106480" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33367 } } peers { permanent_uuid: "d1dd4804029c47089cbf66ab6106c8c5" member_type: NON_VOTER last_known_addr { host: "127.23.100.61" port: 38343 } attrs { promote: true } }
I20260925 10:39:53.095948 30673 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER d1dd4804029c47089cbf66ab6106c8c5 (127.23.100.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a0d0cc835da447eda23faee26d106480" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33367 } } peers { permanent_uuid: "d1dd4804029c47089cbf66ab6106c8c5" member_type: NON_VOTER last_known_addr { host: "127.23.100.61" port: 38343 } attrs { promote: true } } }
I20260925 10:39:53.097071 30618 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:39:53.100337 30618 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5 [term 1 FOLLOWER]: Refusing update from remote peer a0d0cc835da447eda23faee26d106480: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260925 10:39:53.100409 30673 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.23.100.61:38343
I20260925 10:39:53.101046 30673 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a0d0cc835da447eda23faee26d106480" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a0d0cc835da447eda23faee26d106480" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33367 } } peers { permanent_uuid: "d1dd4804029c47089cbf66ab6106c8c5" member_type: NON_VOTER last_known_addr { host: "127.23.100.61" port: 38343 } attrs { promote: true } } }
I20260925 10:39:53.101224 30576 master.cc:537] Master@127.23.100.61:38343 shutting down...
I20260925 10:39:53.101287 30673 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480 [sys.catalog]: This master's current role is: LEADER
I20260925 10:39:53.101476 30672 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1dd4804029c47089cbf66ab6106c8c5" member_type: NON_VOTER last_known_addr { host: "127.23.100.61" port: 38343 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.001s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.100.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/oryx_127.23.100.60.keytab.
Entry for principal oryx/127.23.100.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/oryx_127.23.100.60.keytab.
I20260925 10:39:53.104197 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.60:39145
--webserver_interface=127.23.100.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/oryx_127.23.100.60.keytab
--principal=oryx/127.23.100.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:37749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.62:33367,127.23.100.61:38343,127.23.100.60:39145
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:39:53.105466 30102 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260925 10:39:53.112623 30102 consensus_peers.cc:487] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480 -> Peer d1dd4804029c47089cbf66ab6106c8c5 (127.23.100.61:38343): Couldn't send request to peer d1dd4804029c47089cbf66ab6106c8c5. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 10:39:53.117995 30576 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:39:53.118496 30576 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:39:53.118739 30576 tablet_replica.cc:330] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5: stopping tablet replica
I20260925 10:39:53.136025 30576 master.cc:552] Master@127.23.100.61:38343 shutdown complete.
I20260925 10:39:53.136111 30576 master_runner.cc:305] Clearing existing system tablet
I20260925 10:39:53.136603 30576 fs_manager.cc:359] Metadata directory not provided
I20260925 10:39:53.136726 30576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:39:53.139766 30576 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 10:39:53.141398 30678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:53.141781 30576 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:39:53.142059 30576 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:39:53.142169 30576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-1/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-1/wal
uuid: "d1dd4804029c47089cbf66ab6106c8c5"
format_stamp: "Formatted at 2026-09-25 10:39:52 on dist-test-slave-dxdd"
server_key: "80862cd31aa8377b400a8d20a41f786a"
server_key_iv: "4a5a2dcd8b09d9522819e2d3e12c4a5f"
server_key_version: "encryptionkey@0"
I20260925 10:39:53.146668 30576 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:39:53.153307 30576 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 10:39:53.153496 30576 log.cc:1196] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 10:39:53.153941 30576 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5: Deleting consensus metadata
I20260925 10:39:53.154237 30576 master_runner.cc:315] Copying system tablet from 127.23.100.62:33367
I20260925 10:39:53.156774 30576 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5: tablet copy: Beginning tablet copy session from remote peer at address 127.23.100.62:33367
I20260925 10:39:53.173007 30306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:53.157623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42200 (local address 127.23.100.62:33367)
0925 10:39:53.157823 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:53.157829 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:53.158374 (+ 545us) server_negotiation.cc:407] Connection header received
0925 10:39:53.158591 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:53.158597 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:53.158695 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:53.158812 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:53.159933 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:53.161067 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:53.162991 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:53.163684 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:53.163813 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:53.166259 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:53.166265 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:53.166270 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:53.166328 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:53.169909 (+ 3581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:53.170726 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:53.170732 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:53.170736 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:53.170906 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:53.171354 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:53.171359 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:53.171362 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:53.171727 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:53.171874 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:53.172354 (+ 480us) server_negotiation.cc:299] Negotiation successful
0925 10:39:53.172586 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 10:39:53.173033 30684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:53.157545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:33367 (local address 127.0.0.1:42200)
0925 10:39:53.158220 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:53.158251 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:53.158433 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:53.158856 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:53.158862 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:53.159408 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:53.159753 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:53.159771 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:53.161235 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:53.161240 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:53.162819 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:53.162836 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:53.163852 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:53.163858 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:53.164022 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:53.164741 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:53.164763 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:53.166023 (+ 1260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:53.170046 (+ 4023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:53.170051 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:53.170055 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:53.170599 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:53.171029 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:53.171035 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:53.171040 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:53.171224 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:53.171856 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:53.171861 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:53.172036 (+ 175us) client_negotiation.cc:769] Sending connection context
0925 10:39:53.172188 (+ 152us) client_negotiation.cc:240] Negotiation successful
0925 10:39:53.172538 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":395,"threads_started":1}
I20260925 10:39:53.174213 30135 tablet_copy_service.cc:140] P a0d0cc835da447eda23faee26d106480: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer d1dd4804029c47089cbf66ab6106c8c5 ({username='oryx', principal='oryx/127.23.100.61@KRBTEST.COM'} at 127.0.0.1:42200)
I20260925 10:39:53.174554 30135 tablet_copy_service.cc:161] P a0d0cc835da447eda23faee26d106480: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer d1dd4804029c47089cbf66ab6106c8c5 at {username='oryx', principal='oryx/127.23.100.61@KRBTEST.COM'} at 127.0.0.1:42200: session id = d1dd4804029c47089cbf66ab6106c8c5-00000000000000000000000000000000
I20260925 10:39:53.178938 30135 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 10:39:53.181136 30576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 10:39:53.187985 30576 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5: tablet copy: Starting download of 0 data blocks...
I20260925 10:39:53.188288 30576 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5: tablet copy: Starting download of 1 WAL segments...
I20260925 10:39:53.191174 30576 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 10:39:53.195518 30135 tablet_copy_service.cc:342] P a0d0cc835da447eda23faee26d106480: Request end of tablet copy session d1dd4804029c47089cbf66ab6106c8c5-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.23.100.61@KRBTEST.COM'} at 127.0.0.1:42200
I20260925 10:39:53.195725 30135 tablet_copy_service.cc:433] P a0d0cc835da447eda23faee26d106480: ending tablet copy session d1dd4804029c47089cbf66ab6106c8c5-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer d1dd4804029c47089cbf66ab6106c8c5
I20260925 10:39:53.206390 30576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:39:53.208984 30685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:53.215378 30686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:53.219431 30688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:53.220134 30576 server_base.cc:1034] running on GCE node
I20260925 10:39:53.220486 30576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:39:53.221160 30576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:39:53.222445 30576 hybrid_clock.cc:648] HybridClock initialized: now 1790332793222385 us; error 46 us; skew 500 ppm
I20260925 10:39:53.222700 30576 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:39:53 dist-test-slave-dxdd krb5kdc[30070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332793, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:39:53.224574 30576 init.cc:377] Logged in from keytab as oryx/127.23.100.61@KRBTEST.COM (short username oryx)
I20260925 10:39:53.225934 30576 webserver.cc:460] Webserver started at http://127.23.100.61:45573/ using document root <none> and password file <none>
I20260925 10:39:53.226240 30576 fs_manager.cc:359] Metadata directory not provided
I20260925 10:39:53.226361 30576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:39:53.228683 30576 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260925 10:39:53.230393 30695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:53.230773 30576 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:39:53.230935 30576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-1/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-1/wal
uuid: "d1dd4804029c47089cbf66ab6106c8c5"
format_stamp: "Formatted at 2026-09-25 10:39:52 on dist-test-slave-dxdd"
server_key: "80862cd31aa8377b400a8d20a41f786a"
server_key_iv: "4a5a2dcd8b09d9522819e2d3e12c4a5f"
server_key_version: "encryptionkey@0"
I20260925 10:39:53.231092 30576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:39:53.241801 30576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:39:53.245158 30576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:39:53.277207 30576 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.61:38343
I20260925 10:39:53.277230 30746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.61:38343 every 8 connection(s)
I20260925 10:39:53.278255 30576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-1/data/info.pb
I20260925 10:39:53.285202 30747 sys_catalog.cc:263] Verifying existing consensus state
I20260925 10:39:53.288340 30747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5: Bootstrap starting.
I20260925 10:39:53.301081 30747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:39:53.301535 30747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5: Bootstrap complete.
I20260925 10:39:53.302435 30747 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a0d0cc835da447eda23faee26d106480" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33367 } } peers { permanent_uuid: "d1dd4804029c47089cbf66ab6106c8c5" member_type: NON_VOTER last_known_addr { host: "127.23.100.61" port: 38343 } attrs { promote: true } }
I20260925 10:39:53.302914 30747 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d1dd4804029c47089cbf66ab6106c8c5, State: Initialized, Role: LEARNER
I20260925 10:39:53.303210 30747 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a0d0cc835da447eda23faee26d106480" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33367 } } peers { permanent_uuid: "d1dd4804029c47089cbf66ab6106c8c5" member_type: NON_VOTER last_known_addr { host: "127.23.100.61" port: 38343 } attrs { promote: true } }
I20260925 10:39:53.305486 30750 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a0d0cc835da447eda23faee26d106480" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33367 } } peers { permanent_uuid: "d1dd4804029c47089cbf66ab6106c8c5" member_type: NON_VOTER last_known_addr { host: "127.23.100.61" port: 38343 } attrs { promote: true } } }
I20260925 10:39:53.306573 30750 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5 [sys.catalog]: This master's current role is: LEARNER
I20260925 10:39:53.306630 30747 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:39:53.313702 30761 catalog_manager.cc:1259] Loaded cluster ID: 20e688bc766742f58dcb363b0a466e9f
I20260925 10:39:53.313856 30761 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5: loading cluster ID for follower catalog manager: success
I20260925 10:39:53.316672 30761 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5: acquiring CA information for follower catalog manager: success
I20260925 10:39:53.318521 30761 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 10:39:53.506717 30571 heartbeater.cc:503] Master 127.23.100.62:33367 was elected leader, sending a full tablet report...
W20260925 10:39:53.512179 30676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:39:53.512426 30676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:39:53.512585 30676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:39:53.519950 30676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:39:53.520040 30676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:39:53.520126 30676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:39:53.520212 30676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:39:53.520269 30676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:39:53.529538 30676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:37749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.62:33367,127.23.100.61:38343,127.23.100.60:39145
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/oryx_127.23.100.60.keytab
--principal=oryx/127.23.100.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.60:39145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.100.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:39:53.530821 30676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:39:53.532173 30676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:39:53.543985 30766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:53.544090 30769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:53.544024 30767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:53.544749 30676 server_base.cc:1034] running on GCE node
I20260925 10:39:53.545526 30676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:39:53.546917 30676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:39:53.548110 30676 hybrid_clock.cc:648] HybridClock initialized: now 1790332793548052 us; error 47 us; skew 500 ppm
I20260925 10:39:53.548429 30676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:39:53 dist-test-slave-dxdd krb5kdc[30070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332793, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:39:53.551440 30676 init.cc:377] Logged in from keytab as oryx/127.23.100.60@KRBTEST.COM (short username oryx)
I20260925 10:39:53.552918 30676 webserver.cc:460] Webserver started at http://127.23.100.60:44859/ using document root <none> and password file <none>
I20260925 10:39:53.553570 30676 fs_manager.cc:359] Metadata directory not provided
I20260925 10:39:53.553691 30676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:39:53.553996 30676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:39:53.557034 30676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-2/data/instance:
uuid: "0ef010492c22408bb64ee77cb04de643"
format_stamp: "Formatted at 2026-09-25 10:39:53 on dist-test-slave-dxdd"
server_key: "249a7ce68a22316e8cbadc3e13ed0bdb"
server_key_iv: "ecb642f67788a42e946a075ea24c586c"
server_key_version: "encryptionkey@0"
I20260925 10:39:53.557760 30676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-2/wal/instance:
uuid: "0ef010492c22408bb64ee77cb04de643"
format_stamp: "Formatted at 2026-09-25 10:39:53 on dist-test-slave-dxdd"
server_key: "249a7ce68a22316e8cbadc3e13ed0bdb"
server_key_iv: "ecb642f67788a42e946a075ea24c586c"
server_key_version: "encryptionkey@0"
I20260925 10:39:53.560355 30669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:53.554268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.61:38343 (local address 127.0.0.1:34412)
0925 10:39:53.554462 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:53.554477 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:53.554622 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:53.555359 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:53.555365 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:53.555393 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:39:53.555580 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:53.555589 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:53.556872 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:53.556877 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:53.558238 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:53.558250 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:53.559501 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:53.559506 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:53.559747 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:53.559797 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 10:39:53.559904 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 10:39:53.560049 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 10:39:53.560384 30775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:53.554490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34412 (local address 127.23.100.61:38343)
0925 10:39:53.555056 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:53.555061 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:53.555075 (+ 14us) server_negotiation.cc:407] Connection header received
0925 10:39:53.555132 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:53.555136 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:53.555198 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:53.555315 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:39:53.555753 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:53.556726 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:53.558438 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:53.559293 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:53.559465 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:53.559711 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:53.559941 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 10:39:53.560070 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":332,"threads_started":1}
I20260925 10:39:53.561520 30722 raft_consensus.cc:1203] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260925 10:39:53.562083 30750 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0d0cc835da447eda23faee26d106480. Latest consensus state: current_term: 1 leader_uuid: "a0d0cc835da447eda23faee26d106480" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a0d0cc835da447eda23faee26d106480" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33367 } } peers { permanent_uuid: "d1dd4804029c47089cbf66ab6106c8c5" member_type: NON_VOTER last_known_addr { host: "127.23.100.61" port: 38343 } attrs { promote: true } } }
I20260925 10:39:53.562363 30750 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5 [sys.catalog]: This master's current role is: LEARNER
I20260925 10:39:53.563467 30676 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260925 10:39:53.566583 30777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:53.567770 30676 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260925 10:39:53.567940 30676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-2/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-2/wal
uuid: "0ef010492c22408bb64ee77cb04de643"
format_stamp: "Formatted at 2026-09-25 10:39:53 on dist-test-slave-dxdd"
server_key: "249a7ce68a22316e8cbadc3e13ed0bdb"
server_key_iv: "ecb642f67788a42e946a075ea24c586c"
server_key_version: "encryptionkey@0"
I20260925 10:39:53.568109 30676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:39:53.589003 30676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:39:53.592896 30676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:39:53.593173 30676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:39:53.611228 30676 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.60:39145
I20260925 10:39:53.611259 30828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.60:39145 every 8 connection(s)
I20260925 10:39:53.612586 30676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-2/data/info.pb
I20260925 10:39:53.616111 30829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:39:53.621454 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 30676
I20260925 10:39:53.622751 30829 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33367 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:39:53 dist-test-slave-dxdd krb5kdc[30070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332793, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.60@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM
Sep 25 10:39:53 dist-test-slave-dxdd krb5kdc[30070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.100.60@KRBTEST.COM
I20260925 10:39:53.641006 30306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:53.625214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42202 (local address 127.23.100.62:33367)
0925 10:39:53.625409 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:53.625415 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:53.625848 (+ 433us) server_negotiation.cc:407] Connection header received
0925 10:39:53.626926 (+ 1078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:53.626932 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:53.627011 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:53.627126 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:53.628286 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:53.629006 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:53.630528 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:53.630904 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:53.631034 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:53.635168 (+ 4134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:53.635173 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:53.635176 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:53.635205 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:53.637888 (+ 2683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:53.638783 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:53.638796 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:53.638799 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:53.638921 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:53.639349 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:53.639370 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:53.639373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:53.639649 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:53.639822 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:53.640424 (+ 602us) server_negotiation.cc:299] Negotiation successful
0925 10:39:53.640683 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 10:39:53.642028 30833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:53.625108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:33367 (local address 127.0.0.1:42202)
0925 10:39:53.625716 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:53.625750 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:53.626686 (+ 936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:53.627187 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:53.627193 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:53.627796 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:53.628134 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:53.628149 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:53.629165 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:53.629170 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:53.630387 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:53.630406 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:53.631118 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:53.631125 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:53.631421 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:53.632079 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:53.632100 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:53.634995 (+ 2895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:53.638075 (+ 3080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:53.638089 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:53.638101 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:53.638651 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:53.639037 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:53.639041 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:53.639047 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:53.639231 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:53.639806 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:53.639812 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:53.640059 (+ 247us) client_negotiation.cc:769] Sending connection context
0925 10:39:53.640311 (+ 252us) client_negotiation.cc:240] Negotiation successful
0925 10:39:53.640755 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"mutex_wait_us":43,"thread_start_us":315,"threads_started":1}
I20260925 10:39:53.642542 30832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:53.623767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37646 (local address 127.23.100.60:39145)
0925 10:39:53.624467 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:53.624476 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:53.624512 (+ 36us) server_negotiation.cc:407] Connection header received
0925 10:39:53.626679 (+ 2167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:53.626698 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:53.626925 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:53.627129 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:53.628506 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:53.629689 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:53.630968 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:53.631560 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:53.631704 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:53.636991 (+ 5287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:53.636999 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:53.637008 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:53.637038 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:53.639626 (+ 2588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:53.640279 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:53.640289 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:53.640292 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:53.640437 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:53.640857 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:53.640861 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:53.640864 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:53.641339 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:53.641611 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:53.641836 (+ 225us) server_negotiation.cc:299] Negotiation successful
0925 10:39:53.642079 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":503,"thread_start_us":317,"threads_started":1}
I20260925 10:39:53.644016 30829 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.61" port: 38343 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:39:53 dist-test-slave-dxdd krb5kdc[30070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332793, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.60@KRBTEST.COM for oryx/127.23.100.61@KRBTEST.COM
I20260925 10:39:53.656066 30833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:53.644666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.61:38343 (local address 127.0.0.1:34428)
0925 10:39:53.644823 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:53.644844 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:53.644933 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:53.645319 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:53.645322 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:53.645722 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:53.645859 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:53.645866 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:53.646984 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:53.646987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:53.647811 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:53.647820 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:53.648554 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:53.648558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:53.648709 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:53.649188 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:53.649203 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:53.651021 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:53.653821 (+ 2800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:53.653827 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:53.653831 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:53.654173 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:53.654572 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:53.654578 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:53.654581 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:53.654709 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:53.655188 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:53.655193 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:53.655328 (+ 135us) client_negotiation.cc:769] Sending connection context
0925 10:39:53.655460 (+ 132us) client_negotiation.cc:240] Negotiation successful
0925 10:39:53.655724 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 10:39:53.656077 30775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:53.644839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34428 (local address 127.23.100.61:38343)
0925 10:39:53.645071 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:53.645077 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:53.645102 (+ 25us) server_negotiation.cc:407] Connection header received
0925 10:39:53.645151 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:53.645154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:53.645212 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:53.645289 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:53.645984 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:53.646874 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:53.648054 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:53.648439 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:53.648537 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:53.651210 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:53.651215 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:53.651218 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:53.651246 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:53.653684 (+ 2438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:53.654286 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:53.654292 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:53.654295 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:53.654463 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:53.654810 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:53.654818 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:53.654823 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:53.655076 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:53.655220 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:53.655546 (+ 326us) server_negotiation.cc:299] Negotiation successful
0925 10:39:53.655751 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 10:39:53.657151 30829 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.60" port: 39145 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:39:53 dist-test-slave-dxdd krb5kdc[30070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332793, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.60@KRBTEST.COM for oryx/127.23.100.60@KRBTEST.COM
I20260925 10:39:53.668426 30833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:53.658060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:39145 (local address 127.0.0.1:37656)
0925 10:39:53.658199 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:53.658217 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:53.658335 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:53.658593 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:53.658596 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:53.658921 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:53.659036 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:53.659044 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:53.659951 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:53.659959 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:53.660763 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:53.660772 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:53.661426 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:53.661431 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:53.661648 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:53.662067 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:53.662082 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:53.663914 (+ 1832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:53.666323 (+ 2409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:53.666328 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:53.666332 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:53.666638 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:53.666983 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:53.666987 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:53.666990 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:53.667122 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:53.667560 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:53.667564 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:53.667686 (+ 122us) client_negotiation.cc:769] Sending connection context
0925 10:39:53.667817 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 10:39:53.668051 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 10:39:53.668483 30832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:53.658050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37656 (local address 127.23.100.60:39145)
0925 10:39:53.658199 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:53.658204 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:53.658263 (+ 59us) server_negotiation.cc:407] Connection header received
0925 10:39:53.658420 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:53.658425 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:53.658476 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:53.658578 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:53.659158 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:53.659834 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:53.660893 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:53.661287 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:53.661409 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:53.664079 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:53.664085 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:53.664089 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:53.664123 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:53.666184 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:53.666744 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:53.666751 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:53.666754 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:53.666877 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:53.667227 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:53.667231 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:53.667234 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:53.667454 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:53.667590 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:53.667912 (+ 322us) server_negotiation.cc:299] Negotiation successful
0925 10:39:53.668184 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":99,"server-negotiator.queue_time_us":36}
I20260925 10:39:53.673034 30829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643: Bootstrap starting.
I20260925 10:39:53.675990 30829 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643: Neither blocks nor log segments found. Creating new log.
I20260925 10:39:53.676973 30829 log.cc:824] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643: Log is configured to *not* fsync() on all Append() calls
I20260925 10:39:53.679438 30829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643: No bootstrap required, opened a new log
I20260925 10:39:53.685901 30829 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d0cc835da447eda23faee26d106480" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33367 } } peers { permanent_uuid: "d1dd4804029c47089cbf66ab6106c8c5" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 38343 } } peers { permanent_uuid: "0ef010492c22408bb64ee77cb04de643" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 39145 } }
I20260925 10:39:53.686179 30829 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:39:53.686285 30829 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ef010492c22408bb64ee77cb04de643, State: Initialized, Role: FOLLOWER
I20260925 10:39:53.686902 30829 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d0cc835da447eda23faee26d106480" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33367 } } peers { permanent_uuid: "d1dd4804029c47089cbf66ab6106c8c5" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 38343 } } peers { permanent_uuid: "0ef010492c22408bb64ee77cb04de643" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 39145 } }
I20260925 10:39:53.687757 30835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d0cc835da447eda23faee26d106480" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33367 } } peers { permanent_uuid: "d1dd4804029c47089cbf66ab6106c8c5" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 38343 } } peers { permanent_uuid: "0ef010492c22408bb64ee77cb04de643" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 39145 } } }
I20260925 10:39:53.688057 30835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:39:53.688580 30829 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 10:39:53.696681 30846 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:39:53.696854 30846 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:39:53.698200 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 30176
I20260925 10:39:53.700865 30676 master_runner.cc:428] Detected that this master 0ef010492c22408bb64ee77cb04de643 is joining an existing cluster
I20260925 10:39:53.700969 30676 master_runner.cc:432] Initiating AddMaster RPC to add 127.23.100.60:39145
Sep 25 10:39:53 dist-test-slave-dxdd krb5kdc[30070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.100.1@KRBTEST.COM
I20260925 10:39:53.725090 30848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:53.700021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46558 (local address 127.23.100.1:43887)
0925 10:39:53.701068 (+ 1047us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:53.701076 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:53.701102 (+ 26us) server_negotiation.cc:407] Connection header received
0925 10:39:53.701177 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:53.701187 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:53.701410 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:53.701587 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:53.703274 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:53.704804 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:53.707320 (+ 2516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:53.708186 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:53.708293 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:53.715273 (+ 6980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:53.715286 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:53.715300 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:53.715344 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:53.720088 (+ 4744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:53.721155 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:53.721167 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:53.721173 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:53.721352 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:53.721875 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:53.721883 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:53.721888 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:53.722423 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:53.722787 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:53.724178 (+ 1391us) server_negotiation.cc:299] Negotiation successful
0925 10:39:53.724480 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":881,"thread_start_us":729,"threads_started":1}
I20260925 10:39:53.726758 30299 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:39:53.726796 30835 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:39:53.727164 30835 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d0cc835da447eda23faee26d106480" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33367 } } peers { permanent_uuid: "d1dd4804029c47089cbf66ab6106c8c5" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 38343 } } peers { permanent_uuid: "0ef010492c22408bb64ee77cb04de643" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 39145 } }
I20260925 10:39:53.728919 30835 leader_election.cc:288] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d0cc835da447eda23faee26d106480 (127.23.100.62:33367), d1dd4804029c47089cbf66ab6106c8c5 (127.23.100.61:38343)
I20260925 10:39:53.729738 30125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0ef010492c22408bb64ee77cb04de643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d0cc835da447eda23faee26d106480" is_pre_election: true
I20260925 10:39:53.730101 30832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:53.707163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37658 (local address 127.23.100.60:39145)
0925 10:39:53.707328 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:53.707333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:53.709562 (+ 2229us) server_negotiation.cc:407] Connection header received
0925 10:39:53.709711 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:53.709717 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:53.709816 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:53.709935 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:53.716832 (+ 6897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:53.717932 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:53.719937 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:53.720665 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:53.720768 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:53.723143 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:53.723150 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:53.723158 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:53.723194 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:53.726756 (+ 3562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:53.727662 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:53.727670 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:53.727676 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:53.727861 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:53.728325 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:53.728332 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:53.728337 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:53.728811 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:53.728978 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:53.729510 (+ 532us) server_negotiation.cc:299] Negotiation successful
0925 10:39:53.729706 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":31}
I20260925 10:39:53.730518 30306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:53.706803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42214 (local address 127.23.100.62:33367)
0925 10:39:53.707009 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:53.707015 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:53.710213 (+ 3198us) server_negotiation.cc:407] Connection header received
0925 10:39:53.710484 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:53.710488 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:53.710587 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:53.710660 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:53.711992 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:53.713108 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:53.716120 (+ 3012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:53.716835 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:53.716928 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:53.719561 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:53.719567 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:53.719572 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:53.719609 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:53.723074 (+ 3465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:53.724050 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:53.724057 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:53.724063 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:53.724218 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:53.724720 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:53.724726 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:53.724729 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:53.725103 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:53.725258 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:53.729696 (+ 4438us) server_negotiation.cc:299] Negotiation successful
0925 10:39:53.730006 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 10:39:53.730587 30722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0ef010492c22408bb64ee77cb04de643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1dd4804029c47089cbf66ab6106c8c5" is_pre_election: true
I20260925 10:39:53.730924 30853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:53.705426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:39145 (local address 127.0.0.1:37658)
0925 10:39:53.709471 (+ 4045us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:53.709488 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:53.709622 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:53.710128 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:53.710133 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:53.716373 (+ 6240us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:53.716685 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:53.716698 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:53.718206 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:53.718210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:53.719805 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:53.719818 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:53.721038 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:53.721044 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:53.721254 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:53.721837 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:53.721859 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:53.722951 (+ 1092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:53.726917 (+ 3966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:53.726923 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:53.726926 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:53.727524 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:53.728020 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:53.728027 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:53.728034 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:53.728209 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:53.729065 (+ 856us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:53.729071 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:53.729263 (+ 192us) client_negotiation.cc:769] Sending connection context
0925 10:39:53.730228 (+ 965us) client_negotiation.cc:240] Negotiation successful
0925 10:39:53.730490 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3914,"thread_start_us":9766,"threads_started":1}
I20260925 10:39:53.731827 30854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:53.705455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:33367 (local address 127.0.0.1:42214)
0925 10:39:53.710128 (+ 4673us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:53.710143 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:53.710297 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:53.710852 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:53.710858 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:53.711457 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:53.711825 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:53.711837 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:53.713283 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:53.713289 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:53.715022 (+ 1733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:53.715041 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:53.717224 (+ 2183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:53.717230 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:53.717431 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:53.718102 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:53.718125 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:53.719309 (+ 1184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:53.723218 (+ 3909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:53.723225 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:53.723231 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:53.723797 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:53.724376 (+ 579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:53.724383 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:53.724387 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:53.724586 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:53.725343 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:53.725348 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:53.729367 (+ 4019us) client_negotiation.cc:769] Sending connection context
0925 10:39:53.730696 (+ 1329us) client_negotiation.cc:240] Negotiation successful
0925 10:39:53.731174 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4549,"mutex_wait_us":1,"spinlock_wait_cycles":16640,"thread_start_us":4378,"threads_started":1}
I20260925 10:39:53.733807 30781 leader_election.cc:302] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ef010492c22408bb64ee77cb04de643; no voters: a0d0cc835da447eda23faee26d106480, d1dd4804029c47089cbf66ab6106c8c5
I20260925 10:39:53.734372 30835 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 10:39:53.749503 30853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:53.710619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.61:38343 (local address 127.0.0.1:34434)
0925 10:39:53.731278 (+ 20659us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:53.731296 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:53.731445 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:53.731973 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:53.731978 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:53.732424 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:53.732636 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:53.732650 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:53.738412 (+ 5762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:53.738417 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:53.739908 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:53.739921 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:53.741083 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:53.741090 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:53.741253 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:53.741833 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:53.741855 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:53.742926 (+ 1071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:53.746746 (+ 3820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:53.746752 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:53.746755 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:53.747253 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:53.747686 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:53.747692 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:53.747696 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:53.747860 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:53.748502 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:53.748507 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:53.748664 (+ 157us) client_negotiation.cc:769] Sending connection context
0925 10:39:53.748817 (+ 153us) client_negotiation.cc:240] Negotiation successful
0925 10:39:53.749101 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20535,"thread_start_us":26334,"threads_started":1}
I20260925 10:39:53.749531 30775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:53.710415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34434 (local address 127.23.100.61:38343)
0925 10:39:53.710601 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:53.710606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:53.731363 (+ 20757us) server_negotiation.cc:407] Connection header received
0925 10:39:53.731606 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:53.731612 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:53.731695 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:53.731771 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:53.736824 (+ 5053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:53.738241 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:53.740093 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:53.740749 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:53.740844 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:53.743125 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:53.743132 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:53.743136 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:53.743174 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:53.746580 (+ 3406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:53.747383 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:53.747388 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:53.747392 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:53.747529 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:53.747986 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:53.747990 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:53.747993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:53.748273 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:53.748424 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:53.748987 (+ 563us) server_negotiation.cc:299] Negotiation successful
0925 10:39:53.749203 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Sep 25 10:39:53 dist-test-slave-dxdd krb5kdc[30070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332790, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.62@KRBTEST.COM for oryx/127.23.100.60@KRBTEST.COM
I20260925 10:39:53.756418 30832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:53.741968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37660 (local address 127.23.100.60:39145)
0925 10:39:53.742111 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:53.742116 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:53.742136 (+ 20us) server_negotiation.cc:407] Connection header received
0925 10:39:53.742215 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:53.742219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:53.742342 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:53.742451 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:53.743321 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:53.744414 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:53.746231 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:53.746953 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:53.747086 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:53.751102 (+ 4016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:53.751108 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:53.751113 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:53.751150 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:53.753898 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:53.754600 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:53.754605 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:53.754608 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:53.754722 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:53.755165 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:53.755170 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:53.755173 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:53.755413 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:53.755559 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:53.755902 (+ 343us) server_negotiation.cc:299] Negotiation successful
0925 10:39:53.756083 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 10:39:53.756445 30669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:53.741758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:39145 (local address 127.0.0.1:37660)
0925 10:39:53.741942 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:53.741960 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:53.742055 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:53.742486 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:53.742495 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:53.742917 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:53.743130 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:53.743141 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:53.744570 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:53.744575 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:53.746038 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:53.746054 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:53.747290 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:53.747297 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:53.747459 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:53.748236 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:53.748255 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:53.750933 (+ 2678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:53.754022 (+ 3089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:53.754028 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:53.754032 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:53.754470 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:53.754828 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:53.754833 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:53.754837 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:53.755019 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:53.755525 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:53.755530 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:53.755680 (+ 150us) client_negotiation.cc:769] Sending connection context
0925 10:39:53.755809 (+ 129us) client_negotiation.cc:240] Negotiation successful
0925 10:39:53.756057 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 10:39:53.757941 30114 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.23.100.60:39145: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "0ef010492c22408bb64ee77cb04de643"
member_type: NON_VOTER
last_known_addr {
host: "127.23.100.60"
port: 39145
}
attrs {
promote: true
}
}
dest_uuid: "a0d0cc835da447eda23faee26d106480"
cas_config_opid_index: 5
I20260925 10:39:53.758860 30114 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a0d0cc835da447eda23faee26d106480" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33367 } } peers { permanent_uuid: "d1dd4804029c47089cbf66ab6106c8c5" member_type: NON_VOTER last_known_addr { host: "127.23.100.61" port: 38343 } attrs { promote: true } } peers { permanent_uuid: "0ef010492c22408bb64ee77cb04de643" member_type: NON_VOTER last_known_addr { host: "127.23.100.60" port: 39145 } attrs { promote: true } }
I20260925 10:39:53.760533 30673 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0ef010492c22408bb64ee77cb04de643 (127.23.100.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a0d0cc835da447eda23faee26d106480" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33367 } } peers { permanent_uuid: "d1dd4804029c47089cbf66ab6106c8c5" member_type: NON_VOTER last_known_addr { host: "127.23.100.61" port: 38343 } attrs { promote: true } } peers { permanent_uuid: "0ef010492c22408bb64ee77cb04de643" member_type: NON_VOTER last_known_addr { host: "127.23.100.60" port: 39145 } attrs { promote: true } } }
I20260925 10:39:53.761080 30722 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5 [term 1 LEARNER]: Refusing update from remote peer a0d0cc835da447eda23faee26d106480: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 10:39:53.764968 30673 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.23.100.60:39145
I20260925 10:39:53.765599 30676 master.cc:537] Master@127.23.100.60:39145 shutting down...
I20260925 10:39:53.776448 30676 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260925 10:39:53.776813 30676 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260925 10:39:53.776988 30676 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643: stopping tablet replica
I20260925 10:39:53.777379 30857 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1dd4804029c47089cbf66ab6106c8c5" member_type: NON_VOTER last_known_addr { host: "127.23.100.61" port: 38343 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 10:39:53.783298 30676 master.cc:552] Master@127.23.100.60:39145 shutdown complete.
I20260925 10:39:53.783370 30676 master_runner.cc:305] Clearing existing system tablet
I20260925 10:39:53.783707 30676 fs_manager.cc:359] Metadata directory not provided
I20260925 10:39:53.783789 30676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:39:53.778020 30858 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a0d0cc835da447eda23faee26d106480" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a0d0cc835da447eda23faee26d106480" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33367 } } peers { permanent_uuid: "d1dd4804029c47089cbf66ab6106c8c5" member_type: NON_VOTER last_known_addr { host: "127.23.100.61" port: 38343 } attrs { promote: true } } peers { permanent_uuid: "0ef010492c22408bb64ee77cb04de643" member_type: NON_VOTER last_known_addr { host: "127.23.100.60" port: 39145 } attrs { promote: true } } }
I20260925 10:39:53.785771 30858 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480 [sys.catalog]: This master's current role is: LEADER
I20260925 10:39:53.791853 30669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:53.791314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:39145 (local address 127.0.0.1:37664)
0925 10:39:53.791499 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:53.791666 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.60:39145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 10:39:53.792162 30100 consensus_peers.cc:487] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480 -> Peer 0ef010492c22408bb64ee77cb04de643 (127.23.100.60:39145): Couldn't send request to peer 0ef010492c22408bb64ee77cb04de643. Status: Network error: Client connection negotiation failed: client connection to 127.23.100.60:39145: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 10:39:53.792128 30722 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0ef010492c22408bb64ee77cb04de643 (127.23.100.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a0d0cc835da447eda23faee26d106480" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33367 } } peers { permanent_uuid: "d1dd4804029c47089cbf66ab6106c8c5" member_type: NON_VOTER last_known_addr { host: "127.23.100.61" port: 38343 } attrs { promote: true } } peers { permanent_uuid: "0ef010492c22408bb64ee77cb04de643" member_type: NON_VOTER last_known_addr { host: "127.23.100.60" port: 39145 } attrs { promote: true } } }
I20260925 10:39:53.794554 30676 fs_manager.cc:709] Time spent opening directory manager: real 0.010s user 0.003s sys 0.001s
I20260925 10:39:53.795982 30750 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a0d0cc835da447eda23faee26d106480" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a0d0cc835da447eda23faee26d106480" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33367 } } peers { permanent_uuid: "d1dd4804029c47089cbf66ab6106c8c5" member_type: NON_VOTER last_known_addr { host: "127.23.100.61" port: 38343 } attrs { promote: true } } peers { permanent_uuid: "0ef010492c22408bb64ee77cb04de643" member_type: NON_VOTER last_known_addr { host: "127.23.100.60" port: 39145 } attrs { promote: true } } }
I20260925 10:39:53.796340 30750 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5 [sys.catalog]: This master's current role is: LEARNER
I20260925 10:39:53.797343 30673 raft_consensus.cc:1050] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480: attempting to promote NON_VOTER d1dd4804029c47089cbf66ab6106c8c5 to VOTER
I20260925 10:39:53.798377 30673 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a0d0cc835da447eda23faee26d106480" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33367 } } peers { permanent_uuid: "d1dd4804029c47089cbf66ab6106c8c5" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 38343 } attrs { promote: false } } peers { permanent_uuid: "0ef010492c22408bb64ee77cb04de643" member_type: NON_VOTER last_known_addr { host: "127.23.100.60" port: 39145 } attrs { promote: true } }
I20260925 10:39:53.801368 30721 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5 [term 1 LEARNER]: Refusing update from remote peer a0d0cc835da447eda23faee26d106480: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 10:39:53.801874 30669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:53.801365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:39145 (local address 127.0.0.1:37672)
0925 10:39:53.801510 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:53.801658 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.60:39145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 10:39:53.801945 30857 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1dd4804029c47089cbf66ab6106c8c5" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 38343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 10:39:53.803071 30669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:53.802724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:39145 (local address 127.0.0.1:37680)
0925 10:39:53.802840 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:53.802921 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.60:39145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 10:39:53.803298 30100 consensus_peers.cc:487] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480 -> Peer 0ef010492c22408bb64ee77cb04de643 (127.23.100.60:39145): Couldn't send request to peer 0ef010492c22408bb64ee77cb04de643. Status: Network error: Client connection negotiation failed: client connection to 127.23.100.60:39145: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 10:39:53.805037 30721 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d1dd4804029c47089cbf66ab6106c8c5 (127.23.100.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a0d0cc835da447eda23faee26d106480" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33367 } } peers { permanent_uuid: "d1dd4804029c47089cbf66ab6106c8c5" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 38343 } attrs { promote: false } } peers { permanent_uuid: "0ef010492c22408bb64ee77cb04de643" member_type: NON_VOTER last_known_addr { host: "127.23.100.60" port: 39145 } attrs { promote: true } } }
I20260925 10:39:53.806535 30864 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d1dd4804029c47089cbf66ab6106c8c5 (127.23.100.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a0d0cc835da447eda23faee26d106480" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33367 } } peers { permanent_uuid: "d1dd4804029c47089cbf66ab6106c8c5" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 38343 } attrs { promote: false } } peers { permanent_uuid: "0ef010492c22408bb64ee77cb04de643" member_type: NON_VOTER last_known_addr { host: "127.23.100.60" port: 39145 } attrs { promote: true } } }
I20260925 10:39:53.810542 30750 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a0d0cc835da447eda23faee26d106480" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a0d0cc835da447eda23faee26d106480" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33367 } } peers { permanent_uuid: "d1dd4804029c47089cbf66ab6106c8c5" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 38343 } attrs { promote: false } } peers { permanent_uuid: "0ef010492c22408bb64ee77cb04de643" member_type: NON_VOTER last_known_addr { host: "127.23.100.60" port: 39145 } attrs { promote: true } } }
I20260925 10:39:53.810880 30750 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:39:53.812311 30863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:53.812635 30676 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:39:53.812932 30676 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.011s sys 0.005s
I20260925 10:39:53.813037 30676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-2/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-2/wal
uuid: "0ef010492c22408bb64ee77cb04de643"
format_stamp: "Formatted at 2026-09-25 10:39:53 on dist-test-slave-dxdd"
server_key: "249a7ce68a22316e8cbadc3e13ed0bdb"
server_key_iv: "ecb642f67788a42e946a075ea24c586c"
server_key_version: "encryptionkey@0"
I20260925 10:39:53.814817 30673 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a0d0cc835da447eda23faee26d106480" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a0d0cc835da447eda23faee26d106480" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33367 } } peers { permanent_uuid: "d1dd4804029c47089cbf66ab6106c8c5" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 38343 } attrs { promote: false } } peers { permanent_uuid: "0ef010492c22408bb64ee77cb04de643" member_type: NON_VOTER last_known_addr { host: "127.23.100.60" port: 39145 } attrs { promote: true } } }
I20260925 10:39:53.815024 30673 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480 [sys.catalog]: This master's current role is: LEADER
I20260925 10:39:53.815373 30673 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a0d0cc835da447eda23faee26d106480" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a0d0cc835da447eda23faee26d106480" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33367 } } peers { permanent_uuid: "d1dd4804029c47089cbf66ab6106c8c5" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 38343 } attrs { promote: false } } peers { permanent_uuid: "0ef010492c22408bb64ee77cb04de643" member_type: NON_VOTER last_known_addr { host: "127.23.100.60" port: 39145 } attrs { promote: true } } }
I20260925 10:39:53.815560 30673 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480 [sys.catalog]: This master's current role is: LEADER
I20260925 10:39:53.816984 30676 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:39:53.821486 30676 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 10:39:53.821652 30676 log.cc:1196] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 10:39:53.822104 30676 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643: Deleting consensus metadata
I20260925 10:39:53.822386 30676 master_runner.cc:315] Copying system tablet from 127.23.100.62:33367
I20260925 10:39:53.824748 30676 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643: tablet copy: Beginning tablet copy session from remote peer at address 127.23.100.62:33367
I20260925 10:39:53.841497 30306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:53.825468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42228 (local address 127.23.100.62:33367)
0925 10:39:53.825723 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:53.825732 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:53.830775 (+ 5043us) server_negotiation.cc:407] Connection header received
0925 10:39:53.831001 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:53.831006 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:53.831079 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:53.831191 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:53.832047 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:53.832993 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:53.834236 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:53.834675 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:53.834808 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:53.836445 (+ 1637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:53.836451 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:53.836455 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:53.836494 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:53.839022 (+ 2528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:53.839671 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:53.839678 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:53.839681 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:53.839802 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:53.840185 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:53.840191 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:53.840195 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:53.840425 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:53.840596 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:53.840946 (+ 350us) server_negotiation.cc:299] Negotiation successful
0925 10:39:53.841155 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260925 10:39:53.841539 30870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:53.825477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:33367 (local address 127.0.0.1:42228)
0925 10:39:53.830640 (+ 5163us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:53.830657 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:53.830838 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:53.831287 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:53.831292 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:53.831732 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:53.831901 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:53.831909 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:53.833131 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:53.833138 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:53.834085 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:53.834096 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:53.834822 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:53.834827 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:53.835014 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:53.835605 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:53.835620 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:53.836268 (+ 648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:53.839148 (+ 2880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:53.839156 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:53.839159 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:53.839539 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:53.839919 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:53.839924 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:53.839927 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:53.840070 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:53.840543 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:53.840548 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:53.840713 (+ 165us) client_negotiation.cc:769] Sending connection context
0925 10:39:53.840845 (+ 132us) client_negotiation.cc:240] Negotiation successful
0925 10:39:53.841155 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5045,"thread_start_us":4965,"threads_started":1}
I20260925 10:39:53.842335 30135 tablet_copy_service.cc:140] P a0d0cc835da447eda23faee26d106480: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 0ef010492c22408bb64ee77cb04de643 ({username='oryx', principal='oryx/127.23.100.60@KRBTEST.COM'} at 127.0.0.1:42228)
I20260925 10:39:53.842500 30135 tablet_copy_service.cc:161] P a0d0cc835da447eda23faee26d106480: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 0ef010492c22408bb64ee77cb04de643 at {username='oryx', principal='oryx/127.23.100.60@KRBTEST.COM'} at 127.0.0.1:42228: session id = 0ef010492c22408bb64ee77cb04de643-00000000000000000000000000000000
I20260925 10:39:53.844724 30135 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 10:39:53.846206 30676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 10:39:53.850678 30676 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643: tablet copy: Starting download of 0 data blocks...
I20260925 10:39:53.850934 30676 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643: tablet copy: Starting download of 1 WAL segments...
I20260925 10:39:53.852911 30676 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 10:39:53.856033 30135 tablet_copy_service.cc:342] P a0d0cc835da447eda23faee26d106480: Request end of tablet copy session 0ef010492c22408bb64ee77cb04de643-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.23.100.60@KRBTEST.COM'} at 127.0.0.1:42228
I20260925 10:39:53.856153 30135 tablet_copy_service.cc:433] P a0d0cc835da447eda23faee26d106480: ending tablet copy session 0ef010492c22408bb64ee77cb04de643-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 0ef010492c22408bb64ee77cb04de643
I20260925 10:39:53.865505 30676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:39:53.867940 30871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:53.880963 30872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:53.881886 30676 server_base.cc:1034] running on GCE node
W20260925 10:39:53.882670 30874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:53.883068 30676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:39:53.883729 30676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:39:53.884908 30676 hybrid_clock.cc:648] HybridClock initialized: now 1790332793884871 us; error 35 us; skew 500 ppm
I20260925 10:39:53.885159 30676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:39:53 dist-test-slave-dxdd krb5kdc[30070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332793, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:39:53.887013 30676 init.cc:377] Logged in from keytab as oryx/127.23.100.60@KRBTEST.COM (short username oryx)
I20260925 10:39:53.888401 30676 webserver.cc:460] Webserver started at http://127.23.100.60:44553/ using document root <none> and password file <none>
I20260925 10:39:53.888742 30676 fs_manager.cc:359] Metadata directory not provided
I20260925 10:39:53.888856 30676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:39:53.891480 30676 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 10:39:53.893141 30881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:53.893569 30676 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:39:53.893695 30676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-2/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-2/wal
uuid: "0ef010492c22408bb64ee77cb04de643"
format_stamp: "Formatted at 2026-09-25 10:39:53 on dist-test-slave-dxdd"
server_key: "249a7ce68a22316e8cbadc3e13ed0bdb"
server_key_iv: "ecb642f67788a42e946a075ea24c586c"
server_key_version: "encryptionkey@0"
I20260925 10:39:53.893916 30676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:39:53.902396 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 30176
I20260925 10:39:53.917939 30676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:39:53.921284 30676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:39:53.927098 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.1:43887
--local_ip_for_outbound_sockets=127.23.100.1
--tserver_master_addrs=127.23.100.62:33367
--webserver_port=44391
--webserver_interface=127.23.100.1
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/oryx_127.23.100.1.keytab
--principal=oryx/127.23.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:37749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.23.100.62:33367,127.23.100.61:38343
--tserver_master_addrs=127.23.100.62:33367,127.23.100.61:38343,127.23.100.60:39145 with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:39:53.990634 30676 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.60:39145
I20260925 10:39:53.990681 30933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.60:39145 every 8 connection(s)
I20260925 10:39:53.991652 30676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/master-2/data/info.pb
I20260925 10:39:53.997691 30934 sys_catalog.cc:263] Verifying existing consensus state
I20260925 10:39:54.001407 30934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643: Bootstrap starting.
I20260925 10:39:54.026681 30934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:39:54.027235 30934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643: Bootstrap complete.
I20260925 10:39:54.028489 30934 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a0d0cc835da447eda23faee26d106480" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33367 } } peers { permanent_uuid: "d1dd4804029c47089cbf66ab6106c8c5" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 38343 } attrs { promote: false } } peers { permanent_uuid: "0ef010492c22408bb64ee77cb04de643" member_type: NON_VOTER last_known_addr { host: "127.23.100.60" port: 39145 } attrs { promote: true } }
I20260925 10:39:54.028792 30934 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0ef010492c22408bb64ee77cb04de643, State: Initialized, Role: LEARNER
I20260925 10:39:54.029053 30934 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a0d0cc835da447eda23faee26d106480" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33367 } } peers { permanent_uuid: "d1dd4804029c47089cbf66ab6106c8c5" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 38343 } attrs { promote: false } } peers { permanent_uuid: "0ef010492c22408bb64ee77cb04de643" member_type: NON_VOTER last_known_addr { host: "127.23.100.60" port: 39145 } attrs { promote: true } }
I20260925 10:39:54.030781 30937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a0d0cc835da447eda23faee26d106480" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33367 } } peers { permanent_uuid: "d1dd4804029c47089cbf66ab6106c8c5" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 38343 } attrs { promote: false } } peers { permanent_uuid: "0ef010492c22408bb64ee77cb04de643" member_type: NON_VOTER last_known_addr { host: "127.23.100.60" port: 39145 } attrs { promote: true } } }
I20260925 10:39:54.031188 30937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643 [sys.catalog]: This master's current role is: LEARNER
I20260925 10:39:54.031221 30934 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:39:54.054375 30948 catalog_manager.cc:1259] Loaded cluster ID: 20e688bc766742f58dcb363b0a466e9f
I20260925 10:39:54.054481 30948 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643: loading cluster ID for follower catalog manager: success
I20260925 10:39:54.057446 30948 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643: acquiring CA information for follower catalog manager: success
I20260925 10:39:54.059466 30948 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 10:39:54.299152 30898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:39:54.299430 30898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:39:54.299556 30898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:39:54.306885 30898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:39:54.307030 30898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:39:54.307235 30898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.1
I20260925 10:39:54.317840 30898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:37749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/oryx_127.23.100.1.keytab
--principal=oryx/127.23.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.1:43887
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.1
--webserver_port=44391
--tserver_master_addrs=127.23.100.62:33367,127.23.100.61:38343,127.23.100.60:39145
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.1
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:39:54.319208 30898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:39:54.320498 30898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:39:54.333448 30953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:54.333487 30954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:54.333686 30956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:54.334288 30898 server_base.cc:1034] running on GCE node
I20260925 10:39:54.334906 30898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:39:54.335749 30898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:39:54.336975 30898 hybrid_clock.cc:648] HybridClock initialized: now 1790332794336940 us; error 44 us; skew 500 ppm
I20260925 10:39:54.337318 30898 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:39:54 dist-test-slave-dxdd krb5kdc[30070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332794, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:39:54.340505 30898 init.cc:377] Logged in from keytab as oryx/127.23.100.1@KRBTEST.COM (short username oryx)
I20260925 10:39:54.341866 30898 webserver.cc:460] Webserver started at http://127.23.100.1:44391/ using document root <none> and password file <none>
I20260925 10:39:54.342582 30898 fs_manager.cc:359] Metadata directory not provided
I20260925 10:39:54.342754 30898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:39:54.347715 30898 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260925 10:39:54.350225 30963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:54.351436 30898 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 10:39:54.351614 30898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-0/wal
uuid: "e23f60bcefdf41b1843f6d5f5fcce1fb"
format_stamp: "Formatted at 2026-09-25 10:39:51 on dist-test-slave-dxdd"
server_key: "117395e44da70038a09c0d0aaa45d569"
server_key_iv: "010d9c3612da0708346bd8ad9e6471d0"
server_key_version: "encryptionkey@0"
I20260925 10:39:54.352510 30898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:39:54.361639 30965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:54.356038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37694 (local address 127.23.100.60:39145)
0925 10:39:54.356643 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:54.356647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:54.356663 (+ 16us) server_negotiation.cc:407] Connection header received
0925 10:39:54.356723 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:54.356726 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:54.356789 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:54.356883 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:39:54.357336 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:54.358417 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:54.359855 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:54.360660 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:54.360804 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:54.360994 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:54.361174 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 10:39:54.361306 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":326,"threads_started":1}
I20260925 10:39:54.361642 30964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:54.355816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:39145 (local address 127.0.0.1:37694)
0925 10:39:54.356319 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:54.356340 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:54.356485 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:54.356944 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:54.356949 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:54.356978 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:39:54.357176 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:54.357190 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:54.358553 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:54.358560 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:54.359709 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:54.359721 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:54.360826 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:54.360831 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:54.360995 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:54.361041 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 10:39:54.361151 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 10:39:54.361308 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":288,"threads_started":1}
I20260925 10:39:54.362691 30909 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260925 10:39:54.363207 30937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0d0cc835da447eda23faee26d106480. Latest consensus state: current_term: 1 leader_uuid: "a0d0cc835da447eda23faee26d106480" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a0d0cc835da447eda23faee26d106480" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33367 } } peers { permanent_uuid: "d1dd4804029c47089cbf66ab6106c8c5" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 38343 } attrs { promote: false } } peers { permanent_uuid: "0ef010492c22408bb64ee77cb04de643" member_type: NON_VOTER last_known_addr { host: "127.23.100.60" port: 39145 } attrs { promote: true } } }
I20260925 10:39:54.363456 30937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643 [sys.catalog]: This master's current role is: LEARNER
I20260925 10:39:54.385985 30898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:39:54.389968 30898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:39:54.390264 30898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:39:54.391242 30898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:39:54.392611 30898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:39:54.392735 30898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:54.392858 30898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:39:54.392943 30898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:54.426924 30898 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.1:43887
I20260925 10:39:54.427014 31077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.1:43887 every 8 connection(s)
I20260925 10:39:54.428335 30898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
I20260925 10:39:54.432931 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 30898
I20260925 10:39:54.433328 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 30310
Sep 25 10:39:54 dist-test-slave-dxdd krb5kdc[30070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332794, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.1@KRBTEST.COM for oryx/127.23.100.61@KRBTEST.COM
Sep 25 10:39:54 dist-test-slave-dxdd krb5kdc[30070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332794, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.1@KRBTEST.COM for oryx/127.23.100.60@KRBTEST.COM
I20260925 10:39:54.463282 30965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:54.431281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:49201 (local address 127.23.100.60:39145)
0925 10:39:54.431568 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:54.431578 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:54.437455 (+ 5877us) server_negotiation.cc:407] Connection header received
0925 10:39:54.440620 (+ 3165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:54.440630 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:54.440736 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:54.440827 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:54.444405 (+ 3578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:54.445704 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:54.448877 (+ 3173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:54.449695 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:54.449807 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:54.455607 (+ 5800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:54.455616 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:54.455624 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:54.455673 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:54.459635 (+ 3962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:54.460653 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:54.460661 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:54.460664 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:54.460817 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:54.461335 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:54.461339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:54.461342 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:54.461669 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:54.461796 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:54.462706 (+ 910us) server_negotiation.cc:299] Negotiation successful
0925 10:39:54.462941 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260925 10:39:54.463304 31082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:54.431150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:40367 (local address 127.23.100.61:38343)
0925 10:39:54.431665 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:54.431670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:54.436984 (+ 5314us) server_negotiation.cc:407] Connection header received
0925 10:39:54.440629 (+ 3645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:54.440638 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:54.440769 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:54.440858 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:54.443991 (+ 3133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:54.445422 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:54.448438 (+ 3016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:54.449242 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:54.449360 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:54.455629 (+ 6269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:54.455638 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:54.455644 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:54.455683 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:54.459927 (+ 4244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:54.460749 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:54.460755 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:54.460758 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:54.460900 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:54.461386 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:54.461393 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:54.461399 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:54.461759 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:54.461885 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:54.462690 (+ 805us) server_negotiation.cc:299] Negotiation successful
0925 10:39:54.462899 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":265,"threads_started":1}
I20260925 10:39:54.465507 31083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:54.432345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.61:38343 (local address 127.23.100.1:40367)
0925 10:39:54.436807 (+ 4462us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:54.436869 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:54.440281 (+ 3412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:54.441389 (+ 1108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:54.441402 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:54.442507 (+ 1105us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:54.443773 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:54.443801 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:54.445648 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:54.445656 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:54.448224 (+ 2568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:54.448254 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:54.449758 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:54.449767 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:54.450103 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:54.450927 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:54.450965 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:54.455344 (+ 4379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:54.460045 (+ 4701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:54.460051 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:54.460054 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:54.460587 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:54.461036 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:54.461043 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:54.461046 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:54.461248 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:54.461959 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:54.461965 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:54.462250 (+ 285us) client_negotiation.cc:769] Sending connection context
0925 10:39:54.463538 (+ 1288us) client_negotiation.cc:240] Negotiation successful
0925 10:39:54.464046 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"mutex_wait_us":201,"spinlock_wait_cycles":50432,"thread_start_us":494,"threads_started":1}
I20260925 10:39:54.465544 31084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:54.432909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:39145 (local address 127.23.100.1:49201)
0925 10:39:54.437342 (+ 4433us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:54.437362 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:54.440284 (+ 2922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:54.441387 (+ 1103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:54.441401 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:54.442506 (+ 1105us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:54.443773 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:54.443801 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:54.445944 (+ 2143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:54.445952 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:54.448692 (+ 2740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:54.448710 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:54.450121 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:54.450129 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:54.450875 (+ 746us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:54.451536 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:54.451566 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:54.455344 (+ 3778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:54.459779 (+ 4435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:54.459789 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:54.459804 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:54.460491 (+ 687us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:54.460973 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:54.460977 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:54.460980 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:54.461201 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:54.461877 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:54.461884 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:54.462245 (+ 361us) client_negotiation.cc:769] Sending connection context
0925 10:39:54.463569 (+ 1324us) client_negotiation.cc:240] Negotiation successful
0925 10:39:54.464207 (+ 638us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3582,"mutex_wait_us":958,"spinlock_wait_cycles":41984,"thread_start_us":3317,"threads_started":1}
I20260925 10:39:54.467222 31080 heartbeater.cc:347] Connected to a master server at 127.23.100.61:38343
I20260925 10:39:54.467248 31078 heartbeater.cc:347] Connected to a master server at 127.23.100.60:39145
I20260925 10:39:54.467552 31080 heartbeater.cc:464] Registering TS with master...
I20260925 10:39:54.467531 31078 heartbeater.cc:464] Registering TS with master...
Sep 25 10:39:54 dist-test-slave-dxdd krb5kdc[30070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332794, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.1@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM
I20260925 10:39:54.468653 31080 heartbeater.cc:511] Master 127.23.100.61:38343 requested a full tablet report, sending...
I20260925 10:39:54.468923 31078 heartbeater.cc:511] Master 127.23.100.60:39145 requested a full tablet report, sending...
I20260925 10:39:54.471232 30712 ts_manager.cc:194] Registered new tserver with Master: e23f60bcefdf41b1843f6d5f5fcce1fb (127.23.100.1:43887)
I20260925 10:39:54.471233 30899 ts_manager.cc:194] Registered new tserver with Master: e23f60bcefdf41b1843f6d5f5fcce1fb (127.23.100.1:43887)
Sep 25 10:39:54 dist-test-slave-dxdd krb5kdc[30070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.100.2@KRBTEST.COM
I20260925 10:39:54.475692 31085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:54.433984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:33367 (local address 127.23.100.1:56875)
0925 10:39:54.458470 (+ 24486us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:54.458488 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:54.458596 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:54.460575 (+ 1979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:54.460579 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:54.461086 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:54.461464 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:54.461478 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:54.462932 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:54.462936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:54.464382 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:54.464395 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:54.465541 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:54.465548 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:54.465746 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:54.466473 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:54.466495 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:54.469166 (+ 2671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:54.473124 (+ 3958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:54.473129 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:54.473133 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:54.473590 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:54.474000 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:54.474003 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:54.474006 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:54.474146 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:54.474947 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:54.474950 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:54.475056 (+ 106us) client_negotiation.cc:769] Sending connection context
0925 10:39:54.475172 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 10:39:54.475345 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24361,"thread_start_us":24187,"threads_started":1}
I20260925 10:39:54.475955 31087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:54.434438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:56875 (local address 127.23.100.62:33367)
0925 10:39:54.455878 (+ 21440us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:54.455886 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:54.460065 (+ 4179us) server_negotiation.cc:407] Connection header received
0925 10:39:54.460145 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:54.460151 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:54.460235 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:54.460336 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:54.461635 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:54.462765 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:54.464535 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:54.465137 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:54.465229 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:54.469381 (+ 4152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:54.469388 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:54.469392 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:54.469427 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:54.472993 (+ 3566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:54.473709 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:54.473715 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:54.473721 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:54.473887 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:54.474255 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:54.474260 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:54.474263 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:54.474666 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:54.474818 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:54.475302 (+ 484us) server_negotiation.cc:299] Negotiation successful
0925 10:39:54.475531 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21283,"thread_start_us":21455,"threads_started":1}
I20260925 10:39:54.476675 31079 heartbeater.cc:347] Connected to a master server at 127.23.100.62:33367
I20260925 10:39:54.476804 31079 heartbeater.cc:464] Registering TS with master...
I20260925 10:39:54.477267 31079 heartbeater.cc:511] Master 127.23.100.62:33367 requested a full tablet report, sending...
I20260925 10:39:54.478487 30114 ts_manager.cc:194] Re-registered known tserver with Master: e23f60bcefdf41b1843f6d5f5fcce1fb (127.23.100.1:43887)
I20260925 10:39:54.479343 30114 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.100.1@KRBTEST.COM'} at 127.23.100.1:56875
I20260925 10:39:54.482038 31088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:54.434874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41500 (local address 127.23.100.2:37033)
0925 10:39:54.457098 (+ 22224us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:54.457109 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:54.457907 (+ 798us) server_negotiation.cc:407] Connection header received
0925 10:39:54.458080 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:54.458092 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:54.458387 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:54.458553 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:54.460711 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:54.462208 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:54.464427 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:54.465223 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:54.465338 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:54.472881 (+ 7543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:54.472891 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:54.472902 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:54.472939 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:54.476585 (+ 3646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:54.477460 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:54.477486 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:54.477489 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:54.477676 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:54.478126 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:54.478133 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:54.478138 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:54.478548 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:54.478830 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:54.479912 (+ 1082us) server_negotiation.cc:299] Negotiation successful
0925 10:39:54.481775 (+ 1863us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22041,"thread_start_us":22228,"threads_started":1}
I20260925 10:39:54.483166 30433 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:39:54.582356 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 30310
I20260925 10:39:54.598779 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.2:37033
--local_ip_for_outbound_sockets=127.23.100.2
--tserver_master_addrs=127.23.100.62:33367
--webserver_port=41649
--webserver_interface=127.23.100.2
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/oryx_127.23.100.2.keytab
--principal=oryx/127.23.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:37749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.23.100.62:33367,127.23.100.61:38343
--tserver_master_addrs=127.23.100.62:33367,127.23.100.61:38343,127.23.100.60:39145 with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:39:54.859443 30864 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a0d0cc835da447eda23faee26d106480" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a0d0cc835da447eda23faee26d106480" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33367 } } peers { permanent_uuid: "d1dd4804029c47089cbf66ab6106c8c5" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 38343 } attrs { promote: false } } peers { permanent_uuid: "0ef010492c22408bb64ee77cb04de643" member_type: NON_VOTER last_known_addr { host: "127.23.100.60" port: 39145 } attrs { promote: true } } }
I20260925 10:39:54.859910 30864 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480 [sys.catalog]: This master's current role is: LEADER
I20260925 10:39:54.861205 31091 raft_consensus.cc:1050] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480: attempting to promote NON_VOTER 0ef010492c22408bb64ee77cb04de643 to VOTER
I20260925 10:39:54.861765 31091 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a0d0cc835da447eda23faee26d106480" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33367 } } peers { permanent_uuid: "d1dd4804029c47089cbf66ab6106c8c5" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 38343 } attrs { promote: false } } peers { permanent_uuid: "0ef010492c22408bb64ee77cb04de643" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 39145 } attrs { promote: false } }
I20260925 10:39:54.863844 30909 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643 [term 1 LEARNER]: Refusing update from remote peer a0d0cc835da447eda23faee26d106480: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260925 10:39:54.863855 30722 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5 [term 1 FOLLOWER]: Refusing update from remote peer a0d0cc835da447eda23faee26d106480: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260925 10:39:54.864392 31092 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1dd4804029c47089cbf66ab6106c8c5" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 38343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260925 10:39:54.864882 30864 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ef010492c22408bb64ee77cb04de643" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 39145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260925 10:39:54.868860 31092 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0ef010492c22408bb64ee77cb04de643 (127.23.100.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a0d0cc835da447eda23faee26d106480" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33367 } } peers { permanent_uuid: "d1dd4804029c47089cbf66ab6106c8c5" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 38343 } attrs { promote: false } } peers { permanent_uuid: "0ef010492c22408bb64ee77cb04de643" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 39145 } attrs { promote: false } } }
I20260925 10:39:54.869366 30722 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0ef010492c22408bb64ee77cb04de643 (127.23.100.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a0d0cc835da447eda23faee26d106480" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33367 } } peers { permanent_uuid: "d1dd4804029c47089cbf66ab6106c8c5" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 38343 } attrs { promote: false } } peers { permanent_uuid: "0ef010492c22408bb64ee77cb04de643" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 39145 } attrs { promote: false } } }
I20260925 10:39:54.869505 30909 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0ef010492c22408bb64ee77cb04de643 (127.23.100.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a0d0cc835da447eda23faee26d106480" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33367 } } peers { permanent_uuid: "d1dd4804029c47089cbf66ab6106c8c5" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 38343 } attrs { promote: false } } peers { permanent_uuid: "0ef010492c22408bb64ee77cb04de643" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 39145 } attrs { promote: false } } }
I20260925 10:39:54.873006 30864 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a0d0cc835da447eda23faee26d106480" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a0d0cc835da447eda23faee26d106480" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33367 } } peers { permanent_uuid: "d1dd4804029c47089cbf66ab6106c8c5" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 38343 } attrs { promote: false } } peers { permanent_uuid: "0ef010492c22408bb64ee77cb04de643" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 39145 } attrs { promote: false } } }
I20260925 10:39:54.873308 30864 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480 [sys.catalog]: This master's current role is: LEADER
I20260925 10:39:54.873524 30864 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a0d0cc835da447eda23faee26d106480" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a0d0cc835da447eda23faee26d106480" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33367 } } peers { permanent_uuid: "d1dd4804029c47089cbf66ab6106c8c5" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 38343 } attrs { promote: false } } peers { permanent_uuid: "0ef010492c22408bb64ee77cb04de643" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 39145 } attrs { promote: false } } }
I20260925 10:39:54.873783 30864 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480 [sys.catalog]: This master's current role is: LEADER
I20260925 10:39:54.879599 31099 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a0d0cc835da447eda23faee26d106480" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a0d0cc835da447eda23faee26d106480" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33367 } } peers { permanent_uuid: "d1dd4804029c47089cbf66ab6106c8c5" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 38343 } attrs { promote: false } } peers { permanent_uuid: "0ef010492c22408bb64ee77cb04de643" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 39145 } attrs { promote: false } } }
I20260925 10:39:54.880008 31099 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:39:54.880532 31098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a0d0cc835da447eda23faee26d106480" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a0d0cc835da447eda23faee26d106480" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33367 } } peers { permanent_uuid: "d1dd4804029c47089cbf66ab6106c8c5" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 38343 } attrs { promote: false } } peers { permanent_uuid: "0ef010492c22408bb64ee77cb04de643" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 39145 } attrs { promote: false } } }
I20260925 10:39:54.880815 31098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:39:54.881310 31097 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a0d0cc835da447eda23faee26d106480" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a0d0cc835da447eda23faee26d106480" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33367 } } peers { permanent_uuid: "d1dd4804029c47089cbf66ab6106c8c5" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 38343 } attrs { promote: false } } peers { permanent_uuid: "0ef010492c22408bb64ee77cb04de643" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 39145 } attrs { promote: false } } }
I20260925 10:39:54.881539 31097 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480 [sys.catalog]: This master's current role is: LEADER
W20260925 10:39:54.926147 31090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:39:54.926407 31090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:39:54.926586 31090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:39:54.934830 31090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:39:54.934943 31090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:39:54.935109 31090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.2
I20260925 10:39:54.948248 31090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:37749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/oryx_127.23.100.2.keytab
--principal=oryx/127.23.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.2:37033
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.2
--webserver_port=41649
--tserver_master_addrs=127.23.100.62:33367,127.23.100.61:38343,127.23.100.60:39145
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.2
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:39:54.949519 31090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:39:54.950711 31090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:39:54.963686 31104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:54.963670 31105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:54.963879 31107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:54.965011 31090 server_base.cc:1034] running on GCE node
I20260925 10:39:54.965612 31090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:39:54.966630 31090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:39:54.967842 31090 hybrid_clock.cc:648] HybridClock initialized: now 1790332794967813 us; error 53 us; skew 500 ppm
I20260925 10:39:54.968152 31090 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:39:54 dist-test-slave-dxdd krb5kdc[30070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332794, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:39:54.971136 31090 init.cc:377] Logged in from keytab as oryx/127.23.100.2@KRBTEST.COM (short username oryx)
I20260925 10:39:54.972503 31090 webserver.cc:460] Webserver started at http://127.23.100.2:41649/ using document root <none> and password file <none>
I20260925 10:39:54.973206 31090 fs_manager.cc:359] Metadata directory not provided
I20260925 10:39:54.973356 31090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:39:54.979133 31090 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20260925 10:39:54.981566 31114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:54.983078 31090 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 10:39:54.983242 31090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-1/wal
uuid: "6bb6c395684244c384deb64401dafeba"
format_stamp: "Formatted at 2026-09-25 10:39:51 on dist-test-slave-dxdd"
server_key: "8188abee403cdffef829628b3a0fc882"
server_key_iv: "1540d4d6b998a0e4d076b314a6c94028"
server_key_version: "encryptionkey@0"
I20260925 10:39:54.984467 31090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:39:55.013420 31090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:39:55.017204 31090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:39:55.017426 31090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:39:55.018275 31090 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:39:55.019520 31090 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:39:55.019603 31090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:55.019678 31090 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:39:55.019731 31090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:55.057061 31090 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.2:37033
I20260925 10:39:55.057159 31226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.2:37033 every 8 connection(s)
I20260925 10:39:55.058624 31090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
I20260925 10:39:55.068102 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 31090
I20260925 10:39:55.068372 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 30443
Sep 25 10:39:55 dist-test-slave-dxdd krb5kdc[30070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332794, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.2@KRBTEST.COM for oryx/127.23.100.60@KRBTEST.COM
Sep 25 10:39:55 dist-test-slave-dxdd krb5kdc[30070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332794, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.2@KRBTEST.COM for oryx/127.23.100.61@KRBTEST.COM
I20260925 10:39:55.090850 31232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:55.061822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:39145 (local address 127.23.100.2:37437)
0925 10:39:55.062805 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:55.062861 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:55.065904 (+ 3043us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:55.066766 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:55.066773 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:55.067709 (+ 936us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:55.068594 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:55.068621 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:55.070662 (+ 2041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:55.070670 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:55.073296 (+ 2626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:55.073314 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:55.074961 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:55.074967 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:55.075326 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:55.076315 (+ 989us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:55.076352 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:55.079914 (+ 3562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:55.084941 (+ 5027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:55.084954 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:55.084973 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:55.085774 (+ 801us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:55.086312 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:55.086344 (+ 32us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:55.086350 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:55.086656 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:55.087559 (+ 903us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:55.087571 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:55.088089 (+ 518us) client_negotiation.cc:769] Sending connection context
0925 10:39:55.088428 (+ 339us) client_negotiation.cc:240] Negotiation successful
0925 10:39:55.089060 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"mutex_wait_us":1,"thread_start_us":1104,"threads_started":1}
I20260925 10:39:55.092430 31231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:55.061290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:37437 (local address 127.23.100.60:39145)
0925 10:39:55.061861 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:55.061867 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:55.062985 (+ 1118us) server_negotiation.cc:407] Connection header received
0925 10:39:55.066236 (+ 3251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:55.066241 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:55.066376 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:55.066522 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:55.068836 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:55.070397 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:55.073512 (+ 3115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:55.074390 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:55.074532 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:55.080297 (+ 5765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:55.080307 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:55.080316 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:55.080368 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:55.084741 (+ 4373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:55.085950 (+ 1209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:55.085958 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:55.085964 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:55.086143 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:55.086825 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:55.086833 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:55.086839 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:55.087267 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:55.087439 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:55.091785 (+ 4346us) server_negotiation.cc:299] Negotiation successful
0925 10:39:55.092070 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":262,"threads_started":1}
I20260925 10:39:55.093688 31227 heartbeater.cc:347] Connected to a master server at 127.23.100.60:39145
I20260925 10:39:55.093984 31227 heartbeater.cc:464] Registering TS with master...
I20260925 10:39:55.094808 31234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:55.063190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:47541 (local address 127.23.100.61:38343)
0925 10:39:55.063640 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:55.063644 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:55.066975 (+ 3331us) server_negotiation.cc:407] Connection header received
0925 10:39:55.067052 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:55.067060 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:55.067184 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:55.067307 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:55.077236 (+ 9929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:55.078598 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:55.080127 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:55.080980 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:55.081136 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:55.086191 (+ 5055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:55.086197 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:55.086202 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:55.086262 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:55.090030 (+ 3768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:55.091665 (+ 1635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:55.091688 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:55.091692 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:55.091857 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:55.092324 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:55.092329 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:55.092332 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:55.093018 (+ 686us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:55.093193 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:55.093959 (+ 766us) server_negotiation.cc:299] Negotiation successful
0925 10:39:55.094355 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":247,"threads_started":1}
I20260925 10:39:55.094847 31227 heartbeater.cc:511] Master 127.23.100.60:39145 requested a full tablet report, sending...
I20260925 10:39:55.096366 31233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:55.062708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.61:38343 (local address 127.23.100.2:47541)
0925 10:39:55.066750 (+ 4042us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:55.066768 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:55.066872 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:55.067537 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:55.067545 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:55.068274 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:55.068709 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:55.068726 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:55.078884 (+ 10158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:55.078888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:55.079974 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:55.079988 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:55.081819 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:55.081828 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:55.082047 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:55.083038 (+ 991us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:55.083069 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:55.085962 (+ 2893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:55.090198 (+ 4236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:55.090202 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:55.090206 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:55.090886 (+ 680us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:55.092010 (+ 1124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:55.092016 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:55.092021 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:55.092200 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:55.093260 (+ 1060us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:55.093264 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:55.093432 (+ 168us) client_negotiation.cc:769] Sending connection context
0925 10:39:55.093567 (+ 135us) client_negotiation.cc:240] Negotiation successful
0925 10:39:55.095494 (+ 1927us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3919,"thread_start_us":3762,"threads_started":1}
I20260925 10:39:55.096700 30899 ts_manager.cc:194] Registered new tserver with Master: 6bb6c395684244c384deb64401dafeba (127.23.100.2:37033)
Sep 25 10:39:55 dist-test-slave-dxdd krb5kdc[30070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.100.3@KRBTEST.COM
I20260925 10:39:55.097683 31229 heartbeater.cc:347] Connected to a master server at 127.23.100.61:38343
I20260925 10:39:55.097807 31229 heartbeater.cc:464] Registering TS with master...
I20260925 10:39:55.098259 31229 heartbeater.cc:511] Master 127.23.100.61:38343 requested a full tablet report, sending...
I20260925 10:39:55.099568 30712 ts_manager.cc:194] Registered new tserver with Master: 6bb6c395684244c384deb64401dafeba (127.23.100.2:37033)
Sep 25 10:39:55 dist-test-slave-dxdd krb5kdc[30070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332794, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.2@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM
I20260925 10:39:55.105222 31238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:55.069884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41114 (local address 127.23.100.3:35055)
0925 10:39:55.084069 (+ 14185us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:55.084077 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:55.084109 (+ 32us) server_negotiation.cc:407] Connection header received
0925 10:39:55.084174 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:55.084185 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:55.084466 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:55.086933 (+ 2467us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:55.087194 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:55.088789 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:55.090901 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:55.091757 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:55.098273 (+ 6516us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:55.098648 (+ 375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:55.098661 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:55.098676 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:55.098718 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:55.102391 (+ 3673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:55.103033 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:55.103039 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:55.103042 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:55.103232 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:55.103604 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:55.103612 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:55.103615 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:55.104006 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:55.104316 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:55.104519 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 10:39:55.104912 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13989,"thread_start_us":14694,"threads_started":1}
I20260925 10:39:55.106585 30566 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:39:55.107405 31232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:55.066872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:33367 (local address 127.23.100.2:54691)
0925 10:39:55.091347 (+ 24475us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:55.091371 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:55.091506 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:55.092683 (+ 1177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:55.092687 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:55.093140 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:55.093331 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:55.093341 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:55.095159 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:55.095163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:55.096561 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:55.096574 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:55.097715 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:55.097720 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:55.097947 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:55.098623 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:55.098645 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:55.101002 (+ 2357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:55.104742 (+ 3740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:55.104746 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:55.104749 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:55.105093 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:55.105501 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:55.105506 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:55.105511 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:55.105631 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:55.106213 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:55.106217 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:55.106403 (+ 186us) client_negotiation.cc:769] Sending connection context
0925 10:39:55.106592 (+ 189us) client_negotiation.cc:240] Negotiation successful
0925 10:39:55.106913 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24324,"thread_start_us":25124,"threads_started":1}
I20260925 10:39:55.107474 31236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:55.067040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:54691 (local address 127.23.100.62:33367)
0925 10:39:55.081363 (+ 14323us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:55.081372 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:55.092194 (+ 10822us) server_negotiation.cc:407] Connection header received
0925 10:39:55.092266 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:55.092272 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:55.092372 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:55.092486 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:55.093517 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:55.094995 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:55.096758 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:55.097382 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:55.097505 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:55.101192 (+ 3687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:55.101199 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:55.101206 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:55.101262 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:55.104608 (+ 3346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:55.105212 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:55.105218 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:55.105222 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:55.105383 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:55.105738 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:55.105742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:55.105745 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:55.106058 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:55.106227 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:55.106699 (+ 472us) server_negotiation.cc:299] Negotiation successful
0925 10:39:55.106961 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11529,"thread_start_us":11430,"threads_started":1}
I20260925 10:39:55.108404 31228 heartbeater.cc:347] Connected to a master server at 127.23.100.62:33367
I20260925 10:39:55.108575 31228 heartbeater.cc:464] Registering TS with master...
I20260925 10:39:55.109091 31228 heartbeater.cc:511] Master 127.23.100.62:33367 requested a full tablet report, sending...
I20260925 10:39:55.110234 30114 ts_manager.cc:194] Re-registered known tserver with Master: 6bb6c395684244c384deb64401dafeba (127.23.100.2:37033)
I20260925 10:39:55.111713 30114 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.100.2@KRBTEST.COM'} at 127.23.100.2:54691
I20260925 10:39:55.191504 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 30443
I20260925 10:39:55.207731 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.3:35055
--local_ip_for_outbound_sockets=127.23.100.3
--tserver_master_addrs=127.23.100.62:33367
--webserver_port=46419
--webserver_interface=127.23.100.3
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/oryx_127.23.100.3.keytab
--principal=oryx/127.23.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:37749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.23.100.62:33367,127.23.100.61:38343
--tserver_master_addrs=127.23.100.62:33367,127.23.100.61:38343,127.23.100.60:39145 with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:39:55.481750 31079 heartbeater.cc:503] Master 127.23.100.62:33367 was elected leader, sending a full tablet report...
W20260925 10:39:55.528002 31240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:39:55.528278 31240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:39:55.528438 31240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:39:55.535409 31240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:39:55.535532 31240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:39:55.535691 31240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.3
I20260925 10:39:55.544900 31240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:37749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/krb5kdc/oryx_127.23.100.3.keytab
--principal=oryx/127.23.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.3:35055
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.3
--webserver_port=46419
--tserver_master_addrs=127.23.100.62:33367,127.23.100.61:38343,127.23.100.60:39145
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.3
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:39:55.546093 31240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:39:55.547310 31240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:39:55.560186 31245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:55.560322 31246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:55.560331 31248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:39:55.561661 31240 server_base.cc:1034] running on GCE node
I20260925 10:39:55.562228 31240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:39:55.563100 31240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:39:55.564314 31240 hybrid_clock.cc:648] HybridClock initialized: now 1790332795564283 us; error 43 us; skew 500 ppm
I20260925 10:39:55.564610 31240 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:39:55 dist-test-slave-dxdd krb5kdc[30070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332795, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:39:55.567695 31240 init.cc:377] Logged in from keytab as oryx/127.23.100.3@KRBTEST.COM (short username oryx)
I20260925 10:39:55.568971 31240 webserver.cc:460] Webserver started at http://127.23.100.3:46419/ using document root <none> and password file <none>
I20260925 10:39:55.569543 31240 fs_manager.cc:359] Metadata directory not provided
I20260925 10:39:55.569648 31240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:39:55.574658 31240 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:39:55.577056 31255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:55.578413 31240 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 10:39:55.578558 31240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-2/wal
uuid: "2971592598aa4a4b8f73d4450d8c92cc"
format_stamp: "Formatted at 2026-09-25 10:39:52 on dist-test-slave-dxdd"
server_key: "389fde49a89293496c10784e78fc951c"
server_key_iv: "6f622726b512d53bf61132f5228422cd"
server_key_version: "encryptionkey@0"
I20260925 10:39:55.579423 31240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:39:55.606127 31240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:39:55.610611 31240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:39:55.610839 31240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:39:55.611687 31240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:39:55.612893 31240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:39:55.612974 31240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:55.613051 31240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:39:55.613106 31240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:39:55.644795 31240 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.3:35055
I20260925 10:39:55.644873 31367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.3:35055 every 8 connection(s)
I20260925 10:39:55.646121 31240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
I20260925 10:39:55.647150 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 31240
Sep 25 10:39:55 dist-test-slave-dxdd krb5kdc[30070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332795, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.3@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM
Sep 25 10:39:55 dist-test-slave-dxdd krb5kdc[30070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332795, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.3@KRBTEST.COM for oryx/127.23.100.60@KRBTEST.COM
Sep 25 10:39:55 dist-test-slave-dxdd krb5kdc[30070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332795, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.3@KRBTEST.COM for oryx/127.23.100.61@KRBTEST.COM
I20260925 10:39:55.707019 31376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:55.649674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:33367 (local address 127.23.100.3:54563)
0925 10:39:55.660704 (+ 11030us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:55.660753 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:55.680559 (+ 19806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:55.684100 (+ 3541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:55.684108 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:55.684745 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:55.685107 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:55.685116 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:55.686787 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:55.686791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:55.688290 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:55.688303 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:55.689311 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:55.689317 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:55.689563 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:55.690249 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:55.690270 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:55.693486 (+ 3216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:55.700012 (+ 6526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:55.700022 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:55.700042 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:55.700717 (+ 675us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:55.702826 (+ 2109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:55.702834 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:55.702838 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:55.703020 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:55.705592 (+ 2572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:55.705597 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:55.705766 (+ 169us) client_negotiation.cc:769] Sending connection context
0925 10:39:55.705902 (+ 136us) client_negotiation.cc:240] Negotiation successful
0925 10:39:55.706191 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10728,"mutex_wait_us":20,"thread_start_us":12637,"threads_started":1}
I20260925 10:39:55.707140 31374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:55.649039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:54375 (local address 127.23.100.60:39145)
0925 10:39:55.660123 (+ 11084us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:55.660133 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:55.672019 (+ 11886us) server_negotiation.cc:407] Connection header received
0925 10:39:55.674360 (+ 2341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:55.674369 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:55.674496 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:55.674618 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:55.678085 (+ 3467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:55.679732 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:55.684040 (+ 4308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:55.684914 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:55.685071 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:55.696218 (+ 11147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:55.696224 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:55.696229 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:55.696269 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:55.699825 (+ 3556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:55.701519 (+ 1694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:55.701537 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:55.701540 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:55.701694 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:55.702239 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:55.702243 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:55.702246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:55.702626 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:55.702750 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:55.706522 (+ 3772us) server_negotiation.cc:299] Negotiation successful
0925 10:39:55.706773 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10912,"thread_start_us":19835,"threads_started":1}
I20260925 10:39:55.707568 31375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:55.650251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:39145 (local address 127.23.100.3:54375)
0925 10:39:55.662723 (+ 12472us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:55.662740 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:55.673754 (+ 11014us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:55.675155 (+ 1401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:55.675172 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:55.676673 (+ 1501us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:55.677828 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:55.677860 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:55.680922 (+ 3062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:55.680931 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:55.683845 (+ 2914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:55.683867 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:55.686186 (+ 2319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:55.686193 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:55.686516 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:55.689127 (+ 2611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:55.689160 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:55.695982 (+ 6822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:55.700848 (+ 4866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:55.700853 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:55.700857 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:55.701379 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:55.701842 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:55.701847 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:55.701852 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:55.702090 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:55.703166 (+ 1076us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:55.703176 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:55.703647 (+ 471us) client_negotiation.cc:769] Sending connection context
0925 10:39:55.703964 (+ 317us) client_negotiation.cc:240] Negotiation successful
0925 10:39:55.704505 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12361,"thread_start_us":10042,"threads_started":1}
I20260925 10:39:55.707996 31373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:55.648985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:54563 (local address 127.23.100.62:33367)
0925 10:39:55.664317 (+ 15332us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:55.664326 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:55.664354 (+ 28us) server_negotiation.cc:407] Connection header received
0925 10:39:55.680867 (+ 16513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:55.680876 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:55.681002 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:55.681131 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:55.685321 (+ 4190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:55.686660 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:55.688416 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:55.689122 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:55.689285 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:55.693738 (+ 4453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:55.693748 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:55.693755 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:55.693807 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:55.699137 (+ 5330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:55.702079 (+ 2942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:55.702091 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:55.702097 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:55.702335 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:55.703387 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:55.703396 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:55.703402 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:55.703844 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:55.704030 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:55.706049 (+ 2019us) server_negotiation.cc:299] Negotiation successful
0925 10:39:55.706681 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15165,"thread_start_us":15373,"threads_started":1}
I20260925 10:39:55.708951 31368 heartbeater.cc:347] Connected to a master server at 127.23.100.60:39145
I20260925 10:39:55.709275 31368 heartbeater.cc:464] Registering TS with master...
I20260925 10:39:55.710100 31378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:55.663023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:48261 (local address 127.23.100.61:38343)
0925 10:39:55.671895 (+ 8872us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:55.671903 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:55.685787 (+ 13884us) server_negotiation.cc:407] Connection header received
0925 10:39:55.685974 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:55.685979 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:55.686083 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:55.686193 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:55.687555 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:55.688533 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:55.690692 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:55.691327 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:55.691431 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:55.697593 (+ 6162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:55.697601 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:55.697607 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:55.697659 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:55.701858 (+ 4199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:55.704124 (+ 2266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:55.704133 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:55.704139 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:55.704351 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:55.704808 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:55.704814 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:55.704819 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:55.705267 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:55.705421 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:55.709187 (+ 3766us) server_negotiation.cc:299] Negotiation successful
0925 10:39:55.709551 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8722,"thread_start_us":8887,"threads_started":1}
I20260925 10:39:55.710217 31368 heartbeater.cc:511] Master 127.23.100.60:39145 requested a full tablet report, sending...
I20260925 10:39:55.711314 31377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:55.663012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.61:38343 (local address 127.23.100.3:48261)
0925 10:39:55.685671 (+ 22659us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:55.685687 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:55.685829 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:55.686395 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:55.686399 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:55.687050 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:39:55.687384 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:55.687394 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:55.688752 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:55.688758 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:55.690517 (+ 1759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:55.690531 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:55.691722 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:55.691728 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:55.691927 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:55.692958 (+ 1031us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:39:55.692977 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:39:55.696971 (+ 3994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:39:55.702514 (+ 5543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:55.702519 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:55.702524 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:55.703216 (+ 692us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:55.704508 (+ 1292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:39:55.704515 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:39:55.704519 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:39:55.704677 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:39:55.705489 (+ 812us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:39:55.705496 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:39:55.705666 (+ 170us) client_negotiation.cc:769] Sending connection context
0925 10:39:55.705817 (+ 151us) client_negotiation.cc:240] Negotiation successful
0925 10:39:55.710713 (+ 4896us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22542,"mutex_wait_us":40,"thread_start_us":22420,"threads_started":1}
I20260925 10:39:55.712208 30899 ts_manager.cc:194] Registered new tserver with Master: 2971592598aa4a4b8f73d4450d8c92cc (127.23.100.3:35055)
I20260925 10:39:55.712477 31369 heartbeater.cc:347] Connected to a master server at 127.23.100.62:33367
I20260925 10:39:55.714488 31369 heartbeater.cc:464] Registering TS with master...
I20260925 10:39:55.714918 31369 heartbeater.cc:511] Master 127.23.100.62:33367 requested a full tablet report, sending...
I20260925 10:39:55.715409 31370 heartbeater.cc:347] Connected to a master server at 127.23.100.61:38343
I20260925 10:39:55.715545 31370 heartbeater.cc:464] Registering TS with master...
I20260925 10:39:55.716074 31370 heartbeater.cc:511] Master 127.23.100.61:38343 requested a full tablet report, sending...
I20260925 10:39:55.716964 30114 ts_manager.cc:194] Re-registered known tserver with Master: 2971592598aa4a4b8f73d4450d8c92cc (127.23.100.3:35055)
I20260925 10:39:55.717442 30712 ts_manager.cc:194] Registered new tserver with Master: 2971592598aa4a4b8f73d4450d8c92cc (127.23.100.3:35055)
I20260925 10:39:55.718087 30114 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.100.3@KRBTEST.COM'} at 127.23.100.3:54563
I20260925 10:39:55.722164 31383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:55.687954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42234 (local address 127.23.100.62:33367)
0925 10:39:55.694988 (+ 7034us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:55.694994 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:55.695012 (+ 18us) server_negotiation.cc:407] Connection header received
0925 10:39:55.695059 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:55.695063 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:55.695136 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:55.695268 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:55.701897 (+ 6629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:55.703134 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:55.704898 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:55.705516 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:55.705610 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:55.711101 (+ 5491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:55.711110 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:55.711115 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:55.711153 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:55.718915 (+ 7762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:55.720143 (+ 1228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:55.720148 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:55.720152 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:55.720297 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:55.720740 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:55.720747 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:55.720750 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:55.721004 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:55.721258 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:55.721590 (+ 332us) server_negotiation.cc:299] Negotiation successful
0925 10:39:55.721788 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6893,"thread_start_us":8245,"threads_started":1}
I20260925 10:39:55.728658 31384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:55.688328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37708 (local address 127.23.100.60:39145)
0925 10:39:55.693604 (+ 5276us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:55.693611 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:55.711865 (+ 18254us) server_negotiation.cc:407] Connection header received
0925 10:39:55.711912 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:55.711917 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:55.712017 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:55.712117 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:55.714188 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:55.716017 (+ 1829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:55.718080 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:55.718772 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:55.718863 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:55.722426 (+ 3563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:55.722435 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:55.722442 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:55.722483 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:55.725905 (+ 3422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:55.726652 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:55.726659 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:55.726662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:55.726808 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:55.727201 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:55.727205 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:55.727209 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:55.727532 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:55.727736 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:55.728084 (+ 348us) server_negotiation.cc:299] Negotiation successful
0925 10:39:55.728285 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5146,"thread_start_us":5069,"threads_started":1}
I20260925 10:39:55.731146 31378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:55.698663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34436 (local address 127.23.100.61:38343)
0925 10:39:55.714298 (+ 15635us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:55.714306 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:55.714327 (+ 21us) server_negotiation.cc:407] Connection header received
0925 10:39:55.715394 (+ 1067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:55.715400 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:55.715468 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:55.715558 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:55.717604 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:55.718729 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:55.720897 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:55.721659 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:55.721760 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:55.725203 (+ 3443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:55.725209 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:55.725213 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:55.725247 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:55.728238 (+ 2991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:55.728901 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:55.728906 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:55.728909 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:55.729044 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:55.729467 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:55.729471 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:55.729474 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:55.729826 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:55.730070 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:55.730580 (+ 510us) server_negotiation.cc:299] Negotiation successful
0925 10:39:55.730787 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13083,"thread_start_us":17725,"threads_started":1}
I20260925 10:39:55.726274 30114 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42234:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:39:55.734755 30114 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:39:55.765064 31396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:55.752492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.2:37033 (local address 127.0.0.1:41508)
0925 10:39:55.753054 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:55.753073 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:55.754457 (+ 1384us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:55.758412 (+ 3955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:55.758420 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:55.758457 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:39:55.758839 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:55.758849 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:55.760542 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:55.760548 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:55.762595 (+ 2047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:55.762609 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:55.764148 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:55.764154 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:55.764327 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:55.764360 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 10:39:55.764450 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 10:39:55.764618 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"mutex_wait_us":46,"thread_start_us":285,"threads_started":1}
I20260925 10:39:55.765542 31397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:55.752856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41508 (local address 127.23.100.2:37033)
0925 10:39:55.753376 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:55.753387 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:55.753423 (+ 36us) server_negotiation.cc:407] Connection header received
0925 10:39:55.754979 (+ 1556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:55.754992 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:55.755350 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:55.755615 (+ 265us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:39:55.759007 (+ 3392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:55.760404 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:55.762762 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:55.763930 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:55.764090 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:55.764300 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:55.764533 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 10:39:55.764716 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":260,"threads_started":1}
I20260925 10:39:55.767306 31398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:55.753325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.3:35055 (local address 127.0.0.1:41116)
0925 10:39:55.756281 (+ 2956us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:55.756302 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:55.756430 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:55.759896 (+ 3466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:55.759901 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:55.759926 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:39:55.760263 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:55.760273 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:55.762976 (+ 2703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:55.762981 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:55.765019 (+ 2038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:55.765035 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:55.766691 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:55.766700 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:55.766858 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:55.766882 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 10:39:55.766949 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 10:39:55.767030 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2774,"mutex_wait_us":56,"thread_start_us":1185,"threads_started":1}
I20260925 10:39:55.768265 31401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:55.753827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41116 (local address 127.23.100.3:35055)
0925 10:39:55.759132 (+ 5305us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:55.759145 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:55.759186 (+ 41us) server_negotiation.cc:407] Connection header received
0925 10:39:55.759305 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:55.759316 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:55.759692 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:55.759930 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:39:55.761438 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:55.762851 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:55.765292 (+ 2441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:55.766509 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:55.766610 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:55.766777 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:55.767835 (+ 1058us) server_negotiation.cc:299] Negotiation successful
0925 10:39:55.767982 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5087,"thread_start_us":4837,"threads_started":1}
I20260925 10:39:55.769770 31399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:55.753325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:43887 (local address 127.0.0.1:46562)
0925 10:39:55.753965 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:55.753983 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:55.754112 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:55.761164 (+ 7052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:55.761169 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:55.761191 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:39:55.761864 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:55.761883 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:55.763660 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:55.763666 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:55.767629 (+ 3963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:55.767644 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:55.768934 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:55.768940 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:55.769105 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:55.769140 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 10:39:55.769270 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 10:39:55.769393 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"mutex_wait_us":45,"thread_start_us":2603,"threads_started":1}
I20260925 10:39:55.769781 31400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:55.754280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46562 (local address 127.23.100.1:43887)
0925 10:39:55.760612 (+ 6332us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:55.760618 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:55.760688 (+ 70us) server_negotiation.cc:407] Connection header received
0925 10:39:55.760789 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:55.760796 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:55.760995 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:55.761174 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:39:55.762080 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:55.763513 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:55.767777 (+ 4264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:55.768741 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:55.768886 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:55.769092 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:55.769306 (+ 214us) server_negotiation.cc:299] Negotiation successful
0925 10:39:55.769479 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6165,"thread_start_us":6027,"threads_started":1}
I20260925 10:39:55.769812 31162 tablet_service.cc:1467] Processing CreateTablet for tablet 370bce6d29da4d448501caf0b1b342dc (DEFAULT_TABLE table=test-table [id=48080393f4794663a61015a5c44d8916]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:39:55.771298 31162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 370bce6d29da4d448501caf0b1b342dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:39:55.771740 31303 tablet_service.cc:1467] Processing CreateTablet for tablet 370bce6d29da4d448501caf0b1b342dc (DEFAULT_TABLE table=test-table [id=48080393f4794663a61015a5c44d8916]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:39:55.773113 31303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 370bce6d29da4d448501caf0b1b342dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:39:55.777758 31013 tablet_service.cc:1467] Processing CreateTablet for tablet 370bce6d29da4d448501caf0b1b342dc (DEFAULT_TABLE table=test-table [id=48080393f4794663a61015a5c44d8916]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:39:55.779086 31013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 370bce6d29da4d448501caf0b1b342dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:39:55.787047 31402 tablet_bootstrap.cc:492] T 370bce6d29da4d448501caf0b1b342dc P 6bb6c395684244c384deb64401dafeba: Bootstrap starting.
I20260925 10:39:55.788806 31403 tablet_bootstrap.cc:492] T 370bce6d29da4d448501caf0b1b342dc P 2971592598aa4a4b8f73d4450d8c92cc: Bootstrap starting.
I20260925 10:39:55.791870 31402 tablet_bootstrap.cc:654] T 370bce6d29da4d448501caf0b1b342dc P 6bb6c395684244c384deb64401dafeba: Neither blocks nor log segments found. Creating new log.
I20260925 10:39:55.792364 31403 tablet_bootstrap.cc:654] T 370bce6d29da4d448501caf0b1b342dc P 2971592598aa4a4b8f73d4450d8c92cc: Neither blocks nor log segments found. Creating new log.
I20260925 10:39:55.792994 31402 log.cc:824] T 370bce6d29da4d448501caf0b1b342dc P 6bb6c395684244c384deb64401dafeba: Log is configured to *not* fsync() on all Append() calls
I20260925 10:39:55.793501 31403 log.cc:824] T 370bce6d29da4d448501caf0b1b342dc P 2971592598aa4a4b8f73d4450d8c92cc: Log is configured to *not* fsync() on all Append() calls
I20260925 10:39:55.796188 31402 tablet_bootstrap.cc:492] T 370bce6d29da4d448501caf0b1b342dc P 6bb6c395684244c384deb64401dafeba: No bootstrap required, opened a new log
I20260925 10:39:55.796449 31402 ts_tablet_manager.cc:1397] T 370bce6d29da4d448501caf0b1b342dc P 6bb6c395684244c384deb64401dafeba: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20260925 10:39:55.796721 31403 tablet_bootstrap.cc:492] T 370bce6d29da4d448501caf0b1b342dc P 2971592598aa4a4b8f73d4450d8c92cc: No bootstrap required, opened a new log
I20260925 10:39:55.796957 31403 ts_tablet_manager.cc:1397] T 370bce6d29da4d448501caf0b1b342dc P 2971592598aa4a4b8f73d4450d8c92cc: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260925 10:39:55.796060 31406 tablet_bootstrap.cc:492] T 370bce6d29da4d448501caf0b1b342dc P e23f60bcefdf41b1843f6d5f5fcce1fb: Bootstrap starting.
I20260925 10:39:55.800464 31406 tablet_bootstrap.cc:654] T 370bce6d29da4d448501caf0b1b342dc P e23f60bcefdf41b1843f6d5f5fcce1fb: Neither blocks nor log segments found. Creating new log.
I20260925 10:39:55.801636 31406 log.cc:824] T 370bce6d29da4d448501caf0b1b342dc P e23f60bcefdf41b1843f6d5f5fcce1fb: Log is configured to *not* fsync() on all Append() calls
I20260925 10:39:55.804513 31406 tablet_bootstrap.cc:492] T 370bce6d29da4d448501caf0b1b342dc P e23f60bcefdf41b1843f6d5f5fcce1fb: No bootstrap required, opened a new log
I20260925 10:39:55.804492 31403 raft_consensus.cc:348] T 370bce6d29da4d448501caf0b1b342dc P 2971592598aa4a4b8f73d4450d8c92cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb6c395684244c384deb64401dafeba" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 37033 } } peers { permanent_uuid: "2971592598aa4a4b8f73d4450d8c92cc" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 35055 } } peers { permanent_uuid: "e23f60bcefdf41b1843f6d5f5fcce1fb" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 43887 } }
I20260925 10:39:55.804760 31406 ts_tablet_manager.cc:1397] T 370bce6d29da4d448501caf0b1b342dc P e23f60bcefdf41b1843f6d5f5fcce1fb: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260925 10:39:55.804896 31403 raft_consensus.cc:374] T 370bce6d29da4d448501caf0b1b342dc P 2971592598aa4a4b8f73d4450d8c92cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:39:55.805027 31403 raft_consensus.cc:729] T 370bce6d29da4d448501caf0b1b342dc P 2971592598aa4a4b8f73d4450d8c92cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2971592598aa4a4b8f73d4450d8c92cc, State: Initialized, Role: FOLLOWER
I20260925 10:39:55.805670 31403 consensus_queue.cc:260] T 370bce6d29da4d448501caf0b1b342dc P 2971592598aa4a4b8f73d4450d8c92cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb6c395684244c384deb64401dafeba" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 37033 } } peers { permanent_uuid: "2971592598aa4a4b8f73d4450d8c92cc" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 35055 } } peers { permanent_uuid: "e23f60bcefdf41b1843f6d5f5fcce1fb" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 43887 } }
I20260925 10:39:55.806648 31402 raft_consensus.cc:348] T 370bce6d29da4d448501caf0b1b342dc P 6bb6c395684244c384deb64401dafeba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb6c395684244c384deb64401dafeba" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 37033 } } peers { permanent_uuid: "2971592598aa4a4b8f73d4450d8c92cc" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 35055 } } peers { permanent_uuid: "e23f60bcefdf41b1843f6d5f5fcce1fb" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 43887 } }
I20260925 10:39:55.807042 31402 raft_consensus.cc:374] T 370bce6d29da4d448501caf0b1b342dc P 6bb6c395684244c384deb64401dafeba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:39:55.807134 31402 raft_consensus.cc:729] T 370bce6d29da4d448501caf0b1b342dc P 6bb6c395684244c384deb64401dafeba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bb6c395684244c384deb64401dafeba, State: Initialized, Role: FOLLOWER
I20260925 10:39:55.807754 31402 consensus_queue.cc:260] T 370bce6d29da4d448501caf0b1b342dc P 6bb6c395684244c384deb64401dafeba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb6c395684244c384deb64401dafeba" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 37033 } } peers { permanent_uuid: "2971592598aa4a4b8f73d4450d8c92cc" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 35055 } } peers { permanent_uuid: "e23f60bcefdf41b1843f6d5f5fcce1fb" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 43887 } }
I20260925 10:39:55.810051 31369 heartbeater.cc:503] Master 127.23.100.62:33367 was elected leader, sending a full tablet report...
I20260925 10:39:55.811946 31403 ts_tablet_manager.cc:1428] T 370bce6d29da4d448501caf0b1b342dc P 2971592598aa4a4b8f73d4450d8c92cc: Time spent starting tablet: real 0.015s user 0.011s sys 0.000s
I20260925 10:39:55.815572 31402 ts_tablet_manager.cc:1428] T 370bce6d29da4d448501caf0b1b342dc P 6bb6c395684244c384deb64401dafeba: Time spent starting tablet: real 0.019s user 0.015s sys 0.003s
I20260925 10:39:55.816592 31228 heartbeater.cc:503] Master 127.23.100.62:33367 was elected leader, sending a full tablet report...
I20260925 10:39:55.817106 31406 raft_consensus.cc:348] T 370bce6d29da4d448501caf0b1b342dc P e23f60bcefdf41b1843f6d5f5fcce1fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb6c395684244c384deb64401dafeba" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 37033 } } peers { permanent_uuid: "2971592598aa4a4b8f73d4450d8c92cc" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 35055 } } peers { permanent_uuid: "e23f60bcefdf41b1843f6d5f5fcce1fb" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 43887 } }
I20260925 10:39:55.817523 31406 raft_consensus.cc:374] T 370bce6d29da4d448501caf0b1b342dc P e23f60bcefdf41b1843f6d5f5fcce1fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:39:55.817615 31406 raft_consensus.cc:729] T 370bce6d29da4d448501caf0b1b342dc P e23f60bcefdf41b1843f6d5f5fcce1fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e23f60bcefdf41b1843f6d5f5fcce1fb, State: Initialized, Role: FOLLOWER
I20260925 10:39:55.818172 31406 consensus_queue.cc:260] T 370bce6d29da4d448501caf0b1b342dc P e23f60bcefdf41b1843f6d5f5fcce1fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb6c395684244c384deb64401dafeba" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 37033 } } peers { permanent_uuid: "2971592598aa4a4b8f73d4450d8c92cc" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 35055 } } peers { permanent_uuid: "e23f60bcefdf41b1843f6d5f5fcce1fb" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 43887 } }
I20260925 10:39:55.820091 31406 ts_tablet_manager.cc:1428] T 370bce6d29da4d448501caf0b1b342dc P e23f60bcefdf41b1843f6d5f5fcce1fb: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260925 10:39:55.846638 31408 raft_consensus.cc:482] T 370bce6d29da4d448501caf0b1b342dc P 2971592598aa4a4b8f73d4450d8c92cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:39:55.846920 31408 raft_consensus.cc:504] T 370bce6d29da4d448501caf0b1b342dc P 2971592598aa4a4b8f73d4450d8c92cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb6c395684244c384deb64401dafeba" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 37033 } } peers { permanent_uuid: "2971592598aa4a4b8f73d4450d8c92cc" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 35055 } } peers { permanent_uuid: "e23f60bcefdf41b1843f6d5f5fcce1fb" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 43887 } }
I20260925 10:39:55.848731 31408 leader_election.cc:288] T 370bce6d29da4d448501caf0b1b342dc P 2971592598aa4a4b8f73d4450d8c92cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6bb6c395684244c384deb64401dafeba (127.23.100.2:37033), e23f60bcefdf41b1843f6d5f5fcce1fb (127.23.100.1:43887)
I20260925 10:39:55.856514 31400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:55.849298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:38013 (local address 127.23.100.1:43887)
0925 10:39:55.849454 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:55.849458 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:55.849516 (+ 58us) server_negotiation.cc:407] Connection header received
0925 10:39:55.849695 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:55.849701 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:55.849777 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:55.849872 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:39:55.850459 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:55.851559 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:55.854286 (+ 2727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:55.855542 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:55.855726 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:55.855785 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:55.856140 (+ 355us) server_negotiation.cc:299] Negotiation successful
0925 10:39:55.856275 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 10:39:55.856673 31375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:55.849112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:43887 (local address 127.23.100.3:38013)
0925 10:39:55.849422 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:55.849441 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:55.849555 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:55.849904 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:55.849909 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:55.849942 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:39:55.850287 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:55.850318 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:55.851691 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:55.851697 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:55.854136 (+ 2439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:55.854150 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:55.855698 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:55.855703 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:55.855941 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:55.855977 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 10:39:55.856081 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 10:39:55.856240 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":284,"spinlock_wait_cycles":14336}
I20260925 10:39:55.857067 31377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:55.849069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.2:37033 (local address 127.23.100.3:54415)
0925 10:39:55.849422 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:55.849441 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:55.849555 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:55.849904 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:55.849909 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:55.849939 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:39:55.850288 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:55.850298 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:55.851693 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:55.851697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:55.854136 (+ 2439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:55.854150 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:55.855916 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:55.855920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:55.856143 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:55.856176 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 10:39:55.856273 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 10:39:55.856403 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":265,"spinlock_wait_cycles":14208}
I20260925 10:39:55.857097 31397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:55.849076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:54415 (local address 127.23.100.2:37033)
0925 10:39:55.849270 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:55.849276 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:55.849531 (+ 255us) server_negotiation.cc:407] Connection header received
0925 10:39:55.849707 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:55.849711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:55.849777 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:55.849872 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:39:55.850459 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:55.851558 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:55.854286 (+ 2728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:55.855549 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:55.855709 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:55.855785 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:55.856732 (+ 947us) server_negotiation.cc:299] Negotiation successful
0925 10:39:55.856872 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 10:39:55.857715 31033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370bce6d29da4d448501caf0b1b342dc" candidate_uuid: "2971592598aa4a4b8f73d4450d8c92cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e23f60bcefdf41b1843f6d5f5fcce1fb" is_pre_election: true
I20260925 10:39:55.858071 31182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370bce6d29da4d448501caf0b1b342dc" candidate_uuid: "2971592598aa4a4b8f73d4450d8c92cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bb6c395684244c384deb64401dafeba" is_pre_election: true
I20260925 10:39:55.858218 31033 raft_consensus.cc:2454] T 370bce6d29da4d448501caf0b1b342dc P e23f60bcefdf41b1843f6d5f5fcce1fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2971592598aa4a4b8f73d4450d8c92cc in term 0.
I20260925 10:39:55.858554 31182 raft_consensus.cc:2454] T 370bce6d29da4d448501caf0b1b342dc P 6bb6c395684244c384deb64401dafeba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2971592598aa4a4b8f73d4450d8c92cc in term 0.
I20260925 10:39:55.858970 31258 leader_election.cc:302] T 370bce6d29da4d448501caf0b1b342dc P 2971592598aa4a4b8f73d4450d8c92cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2971592598aa4a4b8f73d4450d8c92cc, e23f60bcefdf41b1843f6d5f5fcce1fb; no voters:
I20260925 10:39:55.859371 31408 raft_consensus.cc:2781] T 370bce6d29da4d448501caf0b1b342dc P 2971592598aa4a4b8f73d4450d8c92cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:39:55.859525 31408 raft_consensus.cc:482] T 370bce6d29da4d448501caf0b1b342dc P 2971592598aa4a4b8f73d4450d8c92cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:39:55.859632 31408 raft_consensus.cc:3037] T 370bce6d29da4d448501caf0b1b342dc P 2971592598aa4a4b8f73d4450d8c92cc [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:39:55.861655 31408 raft_consensus.cc:504] T 370bce6d29da4d448501caf0b1b342dc P 2971592598aa4a4b8f73d4450d8c92cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb6c395684244c384deb64401dafeba" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 37033 } } peers { permanent_uuid: "2971592598aa4a4b8f73d4450d8c92cc" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 35055 } } peers { permanent_uuid: "e23f60bcefdf41b1843f6d5f5fcce1fb" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 43887 } }
I20260925 10:39:55.862193 31408 leader_election.cc:288] T 370bce6d29da4d448501caf0b1b342dc P 2971592598aa4a4b8f73d4450d8c92cc [CANDIDATE]: Term 1 election: Requested vote from peers 6bb6c395684244c384deb64401dafeba (127.23.100.2:37033), e23f60bcefdf41b1843f6d5f5fcce1fb (127.23.100.1:43887)
I20260925 10:39:55.862576 31182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370bce6d29da4d448501caf0b1b342dc" candidate_uuid: "2971592598aa4a4b8f73d4450d8c92cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bb6c395684244c384deb64401dafeba"
I20260925 10:39:55.862764 31182 raft_consensus.cc:3037] T 370bce6d29da4d448501caf0b1b342dc P 6bb6c395684244c384deb64401dafeba [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:39:55.862726 31033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370bce6d29da4d448501caf0b1b342dc" candidate_uuid: "2971592598aa4a4b8f73d4450d8c92cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e23f60bcefdf41b1843f6d5f5fcce1fb"
I20260925 10:39:55.862923 31033 raft_consensus.cc:3037] T 370bce6d29da4d448501caf0b1b342dc P e23f60bcefdf41b1843f6d5f5fcce1fb [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:39:55.864972 31182 raft_consensus.cc:2454] T 370bce6d29da4d448501caf0b1b342dc P 6bb6c395684244c384deb64401dafeba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2971592598aa4a4b8f73d4450d8c92cc in term 1.
I20260925 10:39:55.865124 31033 raft_consensus.cc:2454] T 370bce6d29da4d448501caf0b1b342dc P e23f60bcefdf41b1843f6d5f5fcce1fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2971592598aa4a4b8f73d4450d8c92cc in term 1.
I20260925 10:39:55.865438 31256 leader_election.cc:302] T 370bce6d29da4d448501caf0b1b342dc P 2971592598aa4a4b8f73d4450d8c92cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2971592598aa4a4b8f73d4450d8c92cc, 6bb6c395684244c384deb64401dafeba; no voters:
I20260925 10:39:55.865746 31408 raft_consensus.cc:2781] T 370bce6d29da4d448501caf0b1b342dc P 2971592598aa4a4b8f73d4450d8c92cc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:39:55.866186 31408 raft_consensus.cc:686] T 370bce6d29da4d448501caf0b1b342dc P 2971592598aa4a4b8f73d4450d8c92cc [term 1 LEADER]: Becoming Leader. State: Replica: 2971592598aa4a4b8f73d4450d8c92cc, State: Running, Role: LEADER
I20260925 10:39:55.866765 31408 consensus_queue.cc:237] T 370bce6d29da4d448501caf0b1b342dc P 2971592598aa4a4b8f73d4450d8c92cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb6c395684244c384deb64401dafeba" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 37033 } } peers { permanent_uuid: "2971592598aa4a4b8f73d4450d8c92cc" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 35055 } } peers { permanent_uuid: "e23f60bcefdf41b1843f6d5f5fcce1fb" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 43887 } }
I20260925 10:39:55.871555 30115 catalog_manager.cc:5495] T 370bce6d29da4d448501caf0b1b342dc P 2971592598aa4a4b8f73d4450d8c92cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 2971592598aa4a4b8f73d4450d8c92cc (127.23.100.3). New cstate: current_term: 1 leader_uuid: "2971592598aa4a4b8f73d4450d8c92cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb6c395684244c384deb64401dafeba" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 37033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2971592598aa4a4b8f73d4450d8c92cc" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 35055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e23f60bcefdf41b1843f6d5f5fcce1fb" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 43887 } health_report { overall_health: UNKNOWN } } }
I20260925 10:39:55.891541 31401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:55.885965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41122 (local address 127.23.100.3:35055)
0925 10:39:55.886152 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:55.886160 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:55.886178 (+ 18us) server_negotiation.cc:407] Connection header received
0925 10:39:55.886441 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:55.886444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:55.886519 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:55.886678 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:39:55.887058 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:55.887762 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:55.889187 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:55.889654 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:55.889786 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:55.890274 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:39:55.890960 (+ 686us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:39:55.891070 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:55.891166 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 10:39:55.891231 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 10:39:55.899188 31182 raft_consensus.cc:1261] T 370bce6d29da4d448501caf0b1b342dc P 6bb6c395684244c384deb64401dafeba [term 1 FOLLOWER]: Refusing update from remote peer 2971592598aa4a4b8f73d4450d8c92cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:39:55.899194 31033 raft_consensus.cc:1261] T 370bce6d29da4d448501caf0b1b342dc P e23f60bcefdf41b1843f6d5f5fcce1fb [term 1 FOLLOWER]: Refusing update from remote peer 2971592598aa4a4b8f73d4450d8c92cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:39:55.900104 31411 consensus_queue.cc:1035] T 370bce6d29da4d448501caf0b1b342dc P 2971592598aa4a4b8f73d4450d8c92cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bb6c395684244c384deb64401dafeba" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 37033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:39:55.900466 31408 consensus_queue.cc:1035] T 370bce6d29da4d448501caf0b1b342dc P 2971592598aa4a4b8f73d4450d8c92cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e23f60bcefdf41b1843f6d5f5fcce1fb" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 43887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:39:55.913585 31414 mvcc.cc:205] Tried to move back new op lower bound from 7333203131993866240 to 7333203131873898496. Current Snapshot: MvccSnapshot[applied={T|T < 7333203131993866240}]
I20260925 10:39:55.934672 30115 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:39:55.934942 30115 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:39:55.943758 30115 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a0d0cc835da447eda23faee26d106480: Sending DeleteTablet for 3 replicas of tablet 370bce6d29da4d448501caf0b1b342dc
I20260925 10:39:55.944820 31162 tablet_service.cc:1514] Processing DeleteTablet for tablet 370bce6d29da4d448501caf0b1b342dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:39:55 UTC) from {username='oryx', principal='oryx/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:41508
I20260925 10:39:55.945111 31013 tablet_service.cc:1514] Processing DeleteTablet for tablet 370bce6d29da4d448501caf0b1b342dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:39:55 UTC) from {username='oryx', principal='oryx/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:46562
I20260925 10:39:55.946038 31303 tablet_service.cc:1514] Processing DeleteTablet for tablet 370bce6d29da4d448501caf0b1b342dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:39:55 UTC) from {username='oryx', principal='oryx/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:41116
I20260925 10:39:55.960057 31425 tablet_replica.cc:330] T 370bce6d29da4d448501caf0b1b342dc P e23f60bcefdf41b1843f6d5f5fcce1fb: stopping tablet replica
I20260925 10:39:55.960552 31425 raft_consensus.cc:2229] T 370bce6d29da4d448501caf0b1b342dc P e23f60bcefdf41b1843f6d5f5fcce1fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:39:55.961127 31425 raft_consensus.cc:2258] T 370bce6d29da4d448501caf0b1b342dc P e23f60bcefdf41b1843f6d5f5fcce1fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:39:55.963207 31425 ts_tablet_manager.cc:1905] T 370bce6d29da4d448501caf0b1b342dc P e23f60bcefdf41b1843f6d5f5fcce1fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:39:55.968118 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 30083
I20260925 10:39:55.969342 30145 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:39:55.974586 31425 ts_tablet_manager.cc:1918] T 370bce6d29da4d448501caf0b1b342dc P e23f60bcefdf41b1843f6d5f5fcce1fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:39:55.974758 31425 log.cc:1196] T 370bce6d29da4d448501caf0b1b342dc P e23f60bcefdf41b1843f6d5f5fcce1fb: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-0/wal/wals/370bce6d29da4d448501caf0b1b342dc
I20260925 10:39:55.975415 31425 ts_tablet_manager.cc:1939] T 370bce6d29da4d448501caf0b1b342dc P e23f60bcefdf41b1843f6d5f5fcce1fb: Deleting consensus metadata
I20260925 10:39:55.978885 31426 tablet_replica.cc:330] T 370bce6d29da4d448501caf0b1b342dc P 2971592598aa4a4b8f73d4450d8c92cc: stopping tablet replica
I20260925 10:39:55.979830 31424 tablet_replica.cc:330] T 370bce6d29da4d448501caf0b1b342dc P 6bb6c395684244c384deb64401dafeba: stopping tablet replica
I20260925 10:39:55.980257 31424 raft_consensus.cc:2229] T 370bce6d29da4d448501caf0b1b342dc P 6bb6c395684244c384deb64401dafeba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:39:55.980769 31424 raft_consensus.cc:2258] T 370bce6d29da4d448501caf0b1b342dc P 6bb6c395684244c384deb64401dafeba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:39:55.981204 31426 raft_consensus.cc:2229] T 370bce6d29da4d448501caf0b1b342dc P 2971592598aa4a4b8f73d4450d8c92cc [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:39:55.981941 31426 raft_consensus.cc:2258] T 370bce6d29da4d448501caf0b1b342dc P 2971592598aa4a4b8f73d4450d8c92cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:39:55.984725 31426 ts_tablet_manager.cc:1905] T 370bce6d29da4d448501caf0b1b342dc P 2971592598aa4a4b8f73d4450d8c92cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:39:55.985412 31378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:55.949752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34448 (local address 127.23.100.61:38343)
0925 10:39:55.949952 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:55.949961 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:55.967038 (+ 17077us) server_negotiation.cc:407] Connection header received
0925 10:39:55.967330 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:55.967338 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:55.967425 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:55.967501 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:55.969906 (+ 2405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:55.971103 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:55.973083 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:55.973741 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:55.973851 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:55.977508 (+ 3657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:55.977515 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:55.977520 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:55.977560 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:55.981137 (+ 3577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:55.982998 (+ 1861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:55.983004 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:55.983008 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:55.983161 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:55.983593 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:55.983598 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:55.983602 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:55.984029 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:55.984264 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:55.984803 (+ 539us) server_negotiation.cc:299] Negotiation successful
0925 10:39:55.985082 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 10:39:55.991926 31426 ts_tablet_manager.cc:1918] T 370bce6d29da4d448501caf0b1b342dc P 2971592598aa4a4b8f73d4450d8c92cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:39:55.992121 31426 log.cc:1196] T 370bce6d29da4d448501caf0b1b342dc P 2971592598aa4a4b8f73d4450d8c92cc: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-2/wal/wals/370bce6d29da4d448501caf0b1b342dc
I20260925 10:39:55.992713 31426 ts_tablet_manager.cc:1939] T 370bce6d29da4d448501caf0b1b342dc P 2971592598aa4a4b8f73d4450d8c92cc: Deleting consensus metadata
I20260925 10:39:55.995818 31424 ts_tablet_manager.cc:1905] T 370bce6d29da4d448501caf0b1b342dc P 6bb6c395684244c384deb64401dafeba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:39:56.002947 31424 ts_tablet_manager.cc:1918] T 370bce6d29da4d448501caf0b1b342dc P 6bb6c395684244c384deb64401dafeba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:39:56.003017 31384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:55.967195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37724 (local address 127.23.100.60:39145)
0925 10:39:55.967434 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:55.967442 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:55.986514 (+ 19072us) server_negotiation.cc:407] Connection header received
0925 10:39:55.986783 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:55.986791 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:55.986890 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:55.986973 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:55.988884 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:55.990037 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:55.991940 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:55.992586 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:55.992673 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:55.996329 (+ 3656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:55.996336 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:55.996341 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:55.996384 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:55.999970 (+ 3586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:56.000794 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:56.000800 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:56.000804 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:56.000950 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:56.001356 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:56.001362 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:56.001365 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:56.001698 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:56.001927 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:56.002465 (+ 538us) server_negotiation.cc:299] Negotiation successful
0925 10:39:56.002683 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 10:39:56.003108 31424 log.cc:1196] T 370bce6d29da4d448501caf0b1b342dc P 6bb6c395684244c384deb64401dafeba: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-1/wal/wals/370bce6d29da4d448501caf0b1b342dc
I20260925 10:39:56.003652 31424 ts_tablet_manager.cc:1939] T 370bce6d29da4d448501caf0b1b342dc P 6bb6c395684244c384deb64401dafeba: Deleting consensus metadata
I20260925 10:39:56.130153 30099 catalog_manager.cc:4841] TS 6bb6c395684244c384deb64401dafeba (127.23.100.2:37033): tablet 370bce6d29da4d448501caf0b1b342dc (table test-table [id=48080393f4794663a61015a5c44d8916]) successfully deleted
I20260925 10:39:56.130594 30101 catalog_manager.cc:4841] TS e23f60bcefdf41b1843f6d5f5fcce1fb (127.23.100.1:43887): tablet 370bce6d29da4d448501caf0b1b342dc (table test-table [id=48080393f4794663a61015a5c44d8916]) successfully deleted
I20260925 10:39:56.130846 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 30083
W20260925 10:39:56.130798 30101 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.23.100.1:43887, user_credentials={real_user=slave}} blocked reactor thread for 153368us
I20260925 10:39:56.174506 31378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:56.158523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34450 (local address 127.23.100.61:38343)
0925 10:39:56.158689 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:56.158694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:56.158919 (+ 225us) server_negotiation.cc:407] Connection header received
0925 10:39:56.159142 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:56.159171 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:56.159252 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:56.159358 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:56.161344 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:56.162092 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:56.164179 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:56.164879 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:56.165026 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:56.167882 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:56.167889 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:56.167894 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:56.167938 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:56.171282 (+ 3344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:56.172080 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:56.172086 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:56.172089 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:56.172280 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:56.172722 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:56.172728 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:56.172731 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:56.173090 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:56.173347 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:56.173907 (+ 560us) server_negotiation.cc:299] Negotiation successful
0925 10:39:56.174122 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 10:39:56.176659 31384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:56.156799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37736 (local address 127.23.100.60:39145)
0925 10:39:56.157211 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:56.157217 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:56.158322 (+ 1105us) server_negotiation.cc:407] Connection header received
0925 10:39:56.158568 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:56.158575 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:56.158668 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:56.158747 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:56.161130 (+ 2383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:56.162358 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:56.164318 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:56.165073 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:56.165192 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:56.169815 (+ 4623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:56.169821 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:56.169825 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:56.169866 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:56.173506 (+ 3640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:56.174352 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:56.174357 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:56.174360 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:56.174511 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:56.174966 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:56.174971 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:56.174974 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:56.175335 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:56.175561 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:56.176130 (+ 569us) server_negotiation.cc:299] Negotiation successful
0925 10:39:56.176346 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260925 10:39:56.920089 31435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:56.918819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:33367 (local address 127.23.100.2:36427)
0925 10:39:56.919750 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:56.919889 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:33367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":812,"thread_start_us":683,"threads_started":1}
W20260925 10:39:56.920593 31228 heartbeater.cc:644] Failed to heartbeat to 127.23.100.62:33367 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.100.62:33367: connect: Connection refused (error 111)
I20260925 10:39:56.925544 31436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:56.924432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:33367 (local address 127.23.100.3:57483)
0925 10:39:56.925099 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:56.925250 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:33367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":453,"threads_started":1}
W20260925 10:39:56.925995 31369 heartbeater.cc:644] Failed to heartbeat to 127.23.100.62:33367 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.100.62:33367: connect: Connection refused (error 111)
I20260925 10:39:56.926223 31437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:56.924923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:33367 (local address 127.23.100.1:49947)
0925 10:39:56.925922 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:56.926048 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:33367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":860,"thread_start_us":609,"threads_started":1}
W20260925 10:39:56.926625 31079 heartbeater.cc:644] Failed to heartbeat to 127.23.100.62:33367 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.100.62:33367: connect: Connection refused (error 111)
I20260925 10:39:57.443396 31438 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d0cc835da447eda23faee26d106480)
I20260925 10:39:57.443459 31439 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d0cc835da447eda23faee26d106480)
I20260925 10:39:57.443666 31439 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a0d0cc835da447eda23faee26d106480" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33367 } } peers { permanent_uuid: "d1dd4804029c47089cbf66ab6106c8c5" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 38343 } attrs { promote: false } } peers { permanent_uuid: "0ef010492c22408bb64ee77cb04de643" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 39145 } attrs { promote: false } }
I20260925 10:39:57.443672 31438 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a0d0cc835da447eda23faee26d106480" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33367 } } peers { permanent_uuid: "d1dd4804029c47089cbf66ab6106c8c5" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 38343 } attrs { promote: false } } peers { permanent_uuid: "0ef010492c22408bb64ee77cb04de643" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 39145 } attrs { promote: false } }
I20260925 10:39:57.444443 31439 leader_election.cc:288] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d0cc835da447eda23faee26d106480 (127.23.100.62:33367), d1dd4804029c47089cbf66ab6106c8c5 (127.23.100.61:38343)
I20260925 10:39:57.445209 31438 leader_election.cc:288] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d0cc835da447eda23faee26d106480 (127.23.100.62:33367), 0ef010492c22408bb64ee77cb04de643 (127.23.100.60:39145)
I20260925 10:39:57.445916 31440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:57.444739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:33367 (local address 127.0.0.1:42640)
0925 10:39:57.445258 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:57.445577 (+ 319us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:33367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":231,"threads_started":1}
W20260925 10:39:57.447989 30885 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.100.62:33367: connect: Connection refused (error 111)
I20260925 10:39:57.448499 31441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:57.445335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:33367 (local address 127.0.0.1:42646)
0925 10:39:57.446990 (+ 1655us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:57.448079 (+ 1089us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:33367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1518,"thread_start_us":1366,"threads_started":1}
W20260925 10:39:57.449298 30699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.100.62:33367: connect: Connection refused (error 111)
I20260925 10:39:57.450145 31442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:57.449679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:33367 (local address 127.0.0.1:42656)
0925 10:39:57.449830 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:57.449968 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:33367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 10:39:57.450480 30885 leader_election.cc:334] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a0d0cc835da447eda23faee26d106480 (127.23.100.62:33367): Network error: Client connection negotiation failed: client connection to 127.23.100.62:33367: connect: Connection refused (error 111)
I20260925 10:39:57.452682 31441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:57.452125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:33367 (local address 127.0.0.1:42672)
0925 10:39:57.452409 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:57.452495 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:33367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176}
W20260925 10:39:57.453123 30699 leader_election.cc:334] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a0d0cc835da447eda23faee26d106480 (127.23.100.62:33367): Network error: Client connection negotiation failed: client connection to 127.23.100.62:33367: connect: Connection refused (error 111)
I20260925 10:39:57.455016 31443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:57.445606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:39145 (local address 127.0.0.1:37750)
0925 10:39:57.447336 (+ 1730us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:57.447354 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:57.447491 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:57.448425 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:57.448433 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:57.448466 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:39:57.448893 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:57.448907 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:57.450507 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:57.450512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:57.452747 (+ 2235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:57.452760 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:57.454168 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:57.454173 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:57.454414 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:57.454437 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 10:39:57.454542 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 10:39:57.454663 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1602,"thread_start_us":1387,"threads_started":1}
I20260925 10:39:57.455070 31445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:57.447514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37750 (local address 127.23.100.60:39145)
0925 10:39:57.448006 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:57.448014 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:57.448033 (+ 19us) server_negotiation.cc:407] Connection header received
0925 10:39:57.448099 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:57.448106 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:57.448176 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:57.448290 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:39:57.449166 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:57.450356 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:57.452927 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:57.453990 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:57.454140 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:57.454244 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:57.454566 (+ 322us) server_negotiation.cc:299] Negotiation successful
0925 10:39:57.454688 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":245,"threads_started":1}
I20260925 10:39:57.455495 31444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:57.445606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34458 (local address 127.23.100.61:38343)
0925 10:39:57.447599 (+ 1993us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:57.447604 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:57.447619 (+ 15us) server_negotiation.cc:407] Connection header received
0925 10:39:57.447684 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:57.447687 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:57.447731 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:57.447836 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:39:57.448745 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:57.450221 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:57.452973 (+ 2752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:57.454347 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:57.454477 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:57.454581 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:57.454847 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 10:39:57.454985 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1875,"thread_start_us":2013,"threads_started":1}
I20260925 10:39:57.455757 30909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d1dd4804029c47089cbf66ab6106c8c5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0ef010492c22408bb64ee77cb04de643" is_pre_election: true
I20260925 10:39:57.456084 30909 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1dd4804029c47089cbf66ab6106c8c5 in term 1.
I20260925 10:39:57.456445 31440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:57.445378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.61:38343 (local address 127.0.0.1:34458)
0925 10:39:57.446265 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:57.446285 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:57.446476 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:57.447870 (+ 1394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:57.447877 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:57.447909 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:39:57.448183 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:57.448215 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:57.450413 (+ 2198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:57.450421 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:57.452424 (+ 2003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:57.452437 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:57.454514 (+ 2077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:57.454520 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:57.454687 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:57.454708 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 10:39:57.454828 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 10:39:57.454981 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"thread_start_us":1906,"threads_started":1}
I20260925 10:39:57.456574 30697 leader_election.cc:302] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ef010492c22408bb64ee77cb04de643, d1dd4804029c47089cbf66ab6106c8c5; no voters: a0d0cc835da447eda23faee26d106480
I20260925 10:39:57.456955 31438 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 10:39:57.456979 30722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0ef010492c22408bb64ee77cb04de643" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d1dd4804029c47089cbf66ab6106c8c5" is_pre_election: true
I20260925 10:39:57.457147 31438 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a0d0cc835da447eda23faee26d106480)
I20260925 10:39:57.457255 31438 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 10:39:57.459446 31438 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a0d0cc835da447eda23faee26d106480" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33367 } } peers { permanent_uuid: "d1dd4804029c47089cbf66ab6106c8c5" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 38343 } attrs { promote: false } } peers { permanent_uuid: "0ef010492c22408bb64ee77cb04de643" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 39145 } attrs { promote: false } }
I20260925 10:39:57.459910 30722 raft_consensus.cc:2379] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ef010492c22408bb64ee77cb04de643 in current term 2: Already voted for candidate d1dd4804029c47089cbf66ab6106c8c5 in this term.
I20260925 10:39:57.460171 31438 leader_election.cc:288] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5 [CANDIDATE]: Term 2 election: Requested vote from peers a0d0cc835da447eda23faee26d106480 (127.23.100.62:33367), 0ef010492c22408bb64ee77cb04de643 (127.23.100.60:39145)
I20260925 10:39:57.460645 31443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:57.460205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:33367 (local address 127.0.0.1:42676)
0925 10:39:57.460410 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:57.460509 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:33367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260925 10:39:57.460656 30909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d1dd4804029c47089cbf66ab6106c8c5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0ef010492c22408bb64ee77cb04de643"
I20260925 10:39:57.460908 30909 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643 [term 1 FOLLOWER]: Advancing to term 2
I20260925 10:39:57.461526 31443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:57.461198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:33367 (local address 127.0.0.1:42682)
0925 10:39:57.461329 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:57.461399 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:33367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 10:39:57.461673 30699 leader_election.cc:334] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a0d0cc835da447eda23faee26d106480 (127.23.100.62:33367): Network error: Client connection negotiation failed: client connection to 127.23.100.62:33367: connect: Connection refused (error 111)
I20260925 10:39:57.463102 30909 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1dd4804029c47089cbf66ab6106c8c5 in term 2.
I20260925 10:39:57.463443 30697 leader_election.cc:302] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ef010492c22408bb64ee77cb04de643, d1dd4804029c47089cbf66ab6106c8c5; no voters: a0d0cc835da447eda23faee26d106480
I20260925 10:39:57.463505 30885 leader_election.cc:302] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ef010492c22408bb64ee77cb04de643; no voters: a0d0cc835da447eda23faee26d106480, d1dd4804029c47089cbf66ab6106c8c5
I20260925 10:39:57.463753 31438 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 10:39:57.463812 31439 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 10:39:57.464174 31438 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5 [term 2 LEADER]: Becoming Leader. State: Replica: d1dd4804029c47089cbf66ab6106c8c5, State: Running, Role: LEADER
I20260925 10:39:57.464541 31438 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a0d0cc835da447eda23faee26d106480" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33367 } } peers { permanent_uuid: "d1dd4804029c47089cbf66ab6106c8c5" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 38343 } attrs { promote: false } } peers { permanent_uuid: "0ef010492c22408bb64ee77cb04de643" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 39145 } attrs { promote: false } }
I20260925 10:39:57.466253 31448 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1dd4804029c47089cbf66ab6106c8c5. Latest consensus state: current_term: 2 leader_uuid: "d1dd4804029c47089cbf66ab6106c8c5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a0d0cc835da447eda23faee26d106480" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33367 } } peers { permanent_uuid: "d1dd4804029c47089cbf66ab6106c8c5" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 38343 } attrs { promote: false } } peers { permanent_uuid: "0ef010492c22408bb64ee77cb04de643" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 39145 } attrs { promote: false } } }
I20260925 10:39:57.466658 31448 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5 [sys.catalog]: This master's current role is: LEADER
I20260925 10:39:57.467263 31450 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:39:57.470482 31450 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:39:57.471240 31450 catalog_manager.cc:1259] Loaded cluster ID: 20e688bc766742f58dcb363b0a466e9f
I20260925 10:39:57.471306 31450 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:39:57.472348 31450 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:39:57.473137 31450 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5: Loaded TSK: 0
I20260925 10:39:57.473902 31450 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:39:57.512115 30711 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34450:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:39:57.518755 30711 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:39:57.522531 31443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:57.522117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:33367 (local address 127.0.0.1:42708)
0925 10:39:57.522272 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:57.522396 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:33367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 10:39:57.522588 30909 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643 [term 2 FOLLOWER]: Refusing update from remote peer d1dd4804029c47089cbf66ab6106c8c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260925 10:39:57.523312 31448 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ef010492c22408bb64ee77cb04de643" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 39145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260925 10:39:57.523447 31443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:57.523069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:33367 (local address 127.0.0.1:42710)
0925 10:39:57.523235 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:57.523309 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:33367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 10:39:57.523656 30699 consensus_peers.cc:487] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5 -> Peer a0d0cc835da447eda23faee26d106480 (127.23.100.62:33367): Couldn't send request to peer a0d0cc835da447eda23faee26d106480. Status: Network error: Client connection negotiation failed: client connection to 127.23.100.62:33367: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 10:39:57.526212 31439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1dd4804029c47089cbf66ab6106c8c5. Latest consensus state: current_term: 2 leader_uuid: "d1dd4804029c47089cbf66ab6106c8c5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a0d0cc835da447eda23faee26d106480" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33367 } } peers { permanent_uuid: "d1dd4804029c47089cbf66ab6106c8c5" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 38343 } attrs { promote: false } } peers { permanent_uuid: "0ef010492c22408bb64ee77cb04de643" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 39145 } attrs { promote: false } } }
I20260925 10:39:57.526552 31439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:39:57.527434 31438 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "d1dd4804029c47089cbf66ab6106c8c5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a0d0cc835da447eda23faee26d106480" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33367 } } peers { permanent_uuid: "d1dd4804029c47089cbf66ab6106c8c5" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 38343 } attrs { promote: false } } peers { permanent_uuid: "0ef010492c22408bb64ee77cb04de643" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 39145 } attrs { promote: false } } }
I20260925 10:39:57.527750 31438 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5 [sys.catalog]: This master's current role is: LEADER
I20260925 10:39:57.528019 31439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "d1dd4804029c47089cbf66ab6106c8c5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a0d0cc835da447eda23faee26d106480" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33367 } } peers { permanent_uuid: "d1dd4804029c47089cbf66ab6106c8c5" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 38343 } attrs { promote: false } } peers { permanent_uuid: "0ef010492c22408bb64ee77cb04de643" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 39145 } attrs { promote: false } } }
I20260925 10:39:57.528270 31439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ef010492c22408bb64ee77cb04de643 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:39:57.546816 31441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:57.535585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.3:35055 (local address 127.0.0.1:41136)
0925 10:39:57.535762 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:57.535779 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:57.535885 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:57.539281 (+ 3396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:57.539319 (+ 38us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:57.539345 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:39:57.539933 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:57.539946 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:57.541924 (+ 1978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:57.541931 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:57.544228 (+ 2297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:57.544244 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:57.546016 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:57.546023 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:57.546209 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:57.546250 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 10:39:57.546357 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 10:39:57.546513 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260925 10:39:57.551004 31458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:57.535699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41136 (local address 127.23.100.3:35055)
0925 10:39:57.538709 (+ 3010us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:57.538713 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:57.538728 (+ 15us) server_negotiation.cc:407] Connection header received
0925 10:39:57.538784 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:57.538790 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:57.538843 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:57.538975 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:39:57.540220 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:57.541755 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:57.544475 (+ 2720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:57.545757 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:57.550442 (+ 4685us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:57.550553 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:57.550605 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 10:39:57.550737 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2855,"thread_start_us":2765,"threads_started":1}
I20260925 10:39:57.551666 31303 tablet_service.cc:1467] Processing CreateTablet for tablet ac5c65e3b8b44a7c96fa7e45f277c66a (DEFAULT_TABLE table=test-table [id=11ad5bca444b405b8bc1fac854638450]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:39:57.552182 31303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac5c65e3b8b44a7c96fa7e45f277c66a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:39:57.554281 31441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:57.535694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.2:37033 (local address 127.0.0.1:41510)
0925 10:39:57.547056 (+ 11362us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:57.547074 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:57.547168 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:57.548041 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:57.548045 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:57.548086 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:39:57.548272 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:57.548302 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:57.549767 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:57.549775 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:57.551870 (+ 2095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:57.551887 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:57.553593 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:57.553599 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:57.553778 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:57.553820 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 10:39:57.553903 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 10:39:57.554026 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11239,"thread_start_us":14016,"threads_started":1}
I20260925 10:39:57.554850 31460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:57.536140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41510 (local address 127.23.100.2:37033)
0925 10:39:57.536813 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:57.536822 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:57.547472 (+ 10650us) server_negotiation.cc:407] Connection header received
0925 10:39:57.547582 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:57.547590 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:57.547671 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:57.547769 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:39:57.548476 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:57.549576 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:57.552047 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:57.553242 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:57.553338 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:57.553427 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:57.554416 (+ 989us) server_negotiation.cc:299] Negotiation successful
0925 10:39:57.554558 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":310,"threads_started":1}
I20260925 10:39:57.555824 31162 tablet_service.cc:1467] Processing CreateTablet for tablet ac5c65e3b8b44a7c96fa7e45f277c66a (DEFAULT_TABLE table=test-table [id=11ad5bca444b405b8bc1fac854638450]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:39:57.555976 31443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:57.534726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:43887 (local address 127.0.0.1:46566)
0925 10:39:57.534931 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:57.534949 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:57.535040 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:57.550680 (+ 15640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:57.550685 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:57.550723 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:39:57.550922 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:57.550932 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:57.552267 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:57.552274 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:57.553709 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:57.553722 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:57.555297 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:57.555303 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:57.555506 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:57.555538 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 10:39:57.555632 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 10:39:57.555743 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108}
I20260925 10:39:57.556270 31162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac5c65e3b8b44a7c96fa7e45f277c66a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:39:57.556596 31457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:57.534848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46566 (local address 127.23.100.1:43887)
0925 10:39:57.550191 (+ 15343us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:57.550200 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:57.550221 (+ 21us) server_negotiation.cc:407] Connection header received
0925 10:39:57.550308 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:57.550313 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:57.550422 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:57.551064 (+ 642us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:39:57.551243 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:57.552136 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:57.553852 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:57.555159 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:57.555279 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:57.555388 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:57.556144 (+ 756us) server_negotiation.cc:299] Negotiation successful
0925 10:39:57.556301 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15191,"thread_start_us":15672,"threads_started":1}
I20260925 10:39:57.559005 31461 tablet_bootstrap.cc:492] T ac5c65e3b8b44a7c96fa7e45f277c66a P 2971592598aa4a4b8f73d4450d8c92cc: Bootstrap starting.
I20260925 10:39:57.560710 31461 tablet_bootstrap.cc:654] T ac5c65e3b8b44a7c96fa7e45f277c66a P 2971592598aa4a4b8f73d4450d8c92cc: Neither blocks nor log segments found. Creating new log.
I20260925 10:39:57.561906 31462 tablet_bootstrap.cc:492] T ac5c65e3b8b44a7c96fa7e45f277c66a P 6bb6c395684244c384deb64401dafeba: Bootstrap starting.
I20260925 10:39:57.562153 31013 tablet_service.cc:1467] Processing CreateTablet for tablet ac5c65e3b8b44a7c96fa7e45f277c66a (DEFAULT_TABLE table=test-table [id=11ad5bca444b405b8bc1fac854638450]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:39:57.562580 31013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac5c65e3b8b44a7c96fa7e45f277c66a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:39:57.563035 31461 tablet_bootstrap.cc:492] T ac5c65e3b8b44a7c96fa7e45f277c66a P 2971592598aa4a4b8f73d4450d8c92cc: No bootstrap required, opened a new log
I20260925 10:39:57.563174 31461 ts_tablet_manager.cc:1397] T ac5c65e3b8b44a7c96fa7e45f277c66a P 2971592598aa4a4b8f73d4450d8c92cc: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260925 10:39:57.563753 31462 tablet_bootstrap.cc:654] T ac5c65e3b8b44a7c96fa7e45f277c66a P 6bb6c395684244c384deb64401dafeba: Neither blocks nor log segments found. Creating new log.
I20260925 10:39:57.563987 31461 raft_consensus.cc:348] T ac5c65e3b8b44a7c96fa7e45f277c66a P 2971592598aa4a4b8f73d4450d8c92cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e23f60bcefdf41b1843f6d5f5fcce1fb" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 43887 } } peers { permanent_uuid: "6bb6c395684244c384deb64401dafeba" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 37033 } } peers { permanent_uuid: "2971592598aa4a4b8f73d4450d8c92cc" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 35055 } }
I20260925 10:39:57.564224 31461 raft_consensus.cc:374] T ac5c65e3b8b44a7c96fa7e45f277c66a P 2971592598aa4a4b8f73d4450d8c92cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:39:57.564329 31461 raft_consensus.cc:729] T ac5c65e3b8b44a7c96fa7e45f277c66a P 2971592598aa4a4b8f73d4450d8c92cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2971592598aa4a4b8f73d4450d8c92cc, State: Initialized, Role: FOLLOWER
I20260925 10:39:57.564592 31461 consensus_queue.cc:260] T ac5c65e3b8b44a7c96fa7e45f277c66a P 2971592598aa4a4b8f73d4450d8c92cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e23f60bcefdf41b1843f6d5f5fcce1fb" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 43887 } } peers { permanent_uuid: "6bb6c395684244c384deb64401dafeba" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 37033 } } peers { permanent_uuid: "2971592598aa4a4b8f73d4450d8c92cc" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 35055 } }
I20260925 10:39:57.565418 31461 ts_tablet_manager.cc:1428] T ac5c65e3b8b44a7c96fa7e45f277c66a P 2971592598aa4a4b8f73d4450d8c92cc: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 10:39:57.565913 31462 tablet_bootstrap.cc:492] T ac5c65e3b8b44a7c96fa7e45f277c66a P 6bb6c395684244c384deb64401dafeba: No bootstrap required, opened a new log
I20260925 10:39:57.566037 31462 ts_tablet_manager.cc:1397] T ac5c65e3b8b44a7c96fa7e45f277c66a P 6bb6c395684244c384deb64401dafeba: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260925 10:39:57.566972 31462 raft_consensus.cc:348] T ac5c65e3b8b44a7c96fa7e45f277c66a P 6bb6c395684244c384deb64401dafeba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e23f60bcefdf41b1843f6d5f5fcce1fb" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 43887 } } peers { permanent_uuid: "6bb6c395684244c384deb64401dafeba" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 37033 } } peers { permanent_uuid: "2971592598aa4a4b8f73d4450d8c92cc" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 35055 } }
I20260925 10:39:57.567207 31462 raft_consensus.cc:374] T ac5c65e3b8b44a7c96fa7e45f277c66a P 6bb6c395684244c384deb64401dafeba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:39:57.567317 31462 raft_consensus.cc:729] T ac5c65e3b8b44a7c96fa7e45f277c66a P 6bb6c395684244c384deb64401dafeba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bb6c395684244c384deb64401dafeba, State: Initialized, Role: FOLLOWER
I20260925 10:39:57.567530 31462 consensus_queue.cc:260] T ac5c65e3b8b44a7c96fa7e45f277c66a P 6bb6c395684244c384deb64401dafeba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e23f60bcefdf41b1843f6d5f5fcce1fb" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 43887 } } peers { permanent_uuid: "6bb6c395684244c384deb64401dafeba" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 37033 } } peers { permanent_uuid: "2971592598aa4a4b8f73d4450d8c92cc" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 35055 } }
I20260925 10:39:57.569463 31468 tablet_bootstrap.cc:492] T ac5c65e3b8b44a7c96fa7e45f277c66a P e23f60bcefdf41b1843f6d5f5fcce1fb: Bootstrap starting.
I20260925 10:39:57.570273 31469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:57.569357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:33367 (local address 127.23.100.2:55939)
0925 10:39:57.569841 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:57.570006 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:33367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"threads_started":1}
I20260925 10:39:57.571157 31468 tablet_bootstrap.cc:654] T ac5c65e3b8b44a7c96fa7e45f277c66a P e23f60bcefdf41b1843f6d5f5fcce1fb: Neither blocks nor log segments found. Creating new log.
I20260925 10:39:57.571619 31462 ts_tablet_manager.cc:1428] T ac5c65e3b8b44a7c96fa7e45f277c66a P 6bb6c395684244c384deb64401dafeba: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260925 10:39:57.573493 31469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:57.573059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:33367 (local address 127.23.100.2:54699)
0925 10:39:57.573270 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:57.573352 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:33367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260925 10:39:57.573971 31468 tablet_bootstrap.cc:492] T ac5c65e3b8b44a7c96fa7e45f277c66a P e23f60bcefdf41b1843f6d5f5fcce1fb: No bootstrap required, opened a new log
I20260925 10:39:57.574119 31468 ts_tablet_manager.cc:1397] T ac5c65e3b8b44a7c96fa7e45f277c66a P e23f60bcefdf41b1843f6d5f5fcce1fb: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260925 10:39:57.575105 31466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:57.566078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:33367 (local address 127.23.100.3:47017)
0925 10:39:57.574702 (+ 8624us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:57.574835 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:33367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":8513,"thread_start_us":8374,"threads_started":1}
I20260925 10:39:57.574990 31468 raft_consensus.cc:348] T ac5c65e3b8b44a7c96fa7e45f277c66a P e23f60bcefdf41b1843f6d5f5fcce1fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e23f60bcefdf41b1843f6d5f5fcce1fb" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 43887 } } peers { permanent_uuid: "6bb6c395684244c384deb64401dafeba" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 37033 } } peers { permanent_uuid: "2971592598aa4a4b8f73d4450d8c92cc" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 35055 } }
I20260925 10:39:57.575232 31468 raft_consensus.cc:374] T ac5c65e3b8b44a7c96fa7e45f277c66a P e23f60bcefdf41b1843f6d5f5fcce1fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:39:57.575326 31468 raft_consensus.cc:729] T ac5c65e3b8b44a7c96fa7e45f277c66a P e23f60bcefdf41b1843f6d5f5fcce1fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e23f60bcefdf41b1843f6d5f5fcce1fb, State: Initialized, Role: FOLLOWER
I20260925 10:39:57.575460 31229 heartbeater.cc:503] Master 127.23.100.61:38343 was elected leader, sending a full tablet report...
I20260925 10:39:57.575579 31468 consensus_queue.cc:260] T ac5c65e3b8b44a7c96fa7e45f277c66a P e23f60bcefdf41b1843f6d5f5fcce1fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e23f60bcefdf41b1843f6d5f5fcce1fb" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 43887 } } peers { permanent_uuid: "6bb6c395684244c384deb64401dafeba" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 37033 } } peers { permanent_uuid: "2971592598aa4a4b8f73d4450d8c92cc" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 35055 } }
I20260925 10:39:57.576687 31468 ts_tablet_manager.cc:1428] T ac5c65e3b8b44a7c96fa7e45f277c66a P e23f60bcefdf41b1843f6d5f5fcce1fb: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 10:39:57.578030 31472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:57.577186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:33367 (local address 127.23.100.1:33345)
0925 10:39:57.577639 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:57.577795 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:33367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"threads_started":1}
I20260925 10:39:57.578578 31466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:57.578132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:33367 (local address 127.23.100.3:39819)
0925 10:39:57.578352 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:57.578443 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:33367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260925 10:39:57.578799 31370 heartbeater.cc:503] Master 127.23.100.61:38343 was elected leader, sending a full tablet report...
I20260925 10:39:57.581252 31472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:57.580288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:33367 (local address 127.23.100.1:42409)
0925 10:39:57.581071 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:57.581148 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:33367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":650}
I20260925 10:39:57.605854 31471 raft_consensus.cc:482] T ac5c65e3b8b44a7c96fa7e45f277c66a P e23f60bcefdf41b1843f6d5f5fcce1fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:39:57.606060 31471 raft_consensus.cc:504] T ac5c65e3b8b44a7c96fa7e45f277c66a P e23f60bcefdf41b1843f6d5f5fcce1fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e23f60bcefdf41b1843f6d5f5fcce1fb" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 43887 } } peers { permanent_uuid: "6bb6c395684244c384deb64401dafeba" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 37033 } } peers { permanent_uuid: "2971592598aa4a4b8f73d4450d8c92cc" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 35055 } }
I20260925 10:39:57.607658 31471 leader_election.cc:288] T ac5c65e3b8b44a7c96fa7e45f277c66a P e23f60bcefdf41b1843f6d5f5fcce1fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6bb6c395684244c384deb64401dafeba (127.23.100.2:37033), 2971592598aa4a4b8f73d4450d8c92cc (127.23.100.3:35055)
I20260925 10:39:57.613509 31458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:57.608075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:51553 (local address 127.23.100.3:35055)
0925 10:39:57.608292 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:57.608300 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:57.608445 (+ 145us) server_negotiation.cc:407] Connection header received
0925 10:39:57.608709 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:57.608717 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:57.608798 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:57.608921 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:39:57.609370 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:57.610135 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:57.611865 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:57.612589 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:57.612759 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:57.612840 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:57.613129 (+ 289us) server_negotiation.cc:299] Negotiation successful
0925 10:39:57.613256 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 10:39:57.613552 31472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:57.607864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.3:35055 (local address 127.23.100.1:51553)
0925 10:39:57.608235 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:57.608260 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:57.608536 (+ 276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:57.608943 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:57.608951 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:57.608994 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:39:57.609221 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:57.609228 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:57.610261 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:57.610267 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:57.611702 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:57.611713 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:57.612766 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:57.612771 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:57.612944 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:57.612967 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 10:39:57.613103 (+ 136us) client_negotiation.cc:240] Negotiation successful
0925 10:39:57.613247 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":24}
I20260925 10:39:57.614009 31460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:57.608048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:41747 (local address 127.23.100.2:37033)
0925 10:39:57.608263 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:57.608270 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:57.608947 (+ 677us) server_negotiation.cc:407] Connection header received
0925 10:39:57.609229 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:57.609241 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:57.609298 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:57.609388 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:39:57.609908 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:57.610658 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:57.612320 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:57.613061 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:57.613254 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:57.613324 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:57.613641 (+ 317us) server_negotiation.cc:299] Negotiation successful
0925 10:39:57.613786 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 10:39:57.614286 31323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5c65e3b8b44a7c96fa7e45f277c66a" candidate_uuid: "e23f60bcefdf41b1843f6d5f5fcce1fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2971592598aa4a4b8f73d4450d8c92cc" is_pre_election: true
I20260925 10:39:57.614662 31473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:57.607910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.2:37033 (local address 127.23.100.1:41747)
0925 10:39:57.608857 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:57.608876 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:39:57.609065 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:39:57.609460 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:39:57.609465 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:39:57.609491 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:39:57.609777 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:57.609784 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:57.610790 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:57.610796 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:57.612171 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:39:57.612182 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:39:57.613277 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:39:57.613284 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:39:57.613448 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:57.613471 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 10:39:57.614192 (+ 721us) client_negotiation.cc:240] Negotiation successful
0925 10:39:57.614381 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":829,"mutex_wait_us":42,"spinlock_wait_cycles":1280,"thread_start_us":599,"threads_started":1}
I20260925 10:39:57.614763 31323 raft_consensus.cc:2454] T ac5c65e3b8b44a7c96fa7e45f277c66a P 2971592598aa4a4b8f73d4450d8c92cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e23f60bcefdf41b1843f6d5f5fcce1fb in term 0.
I20260925 10:39:57.615249 31182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5c65e3b8b44a7c96fa7e45f277c66a" candidate_uuid: "e23f60bcefdf41b1843f6d5f5fcce1fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bb6c395684244c384deb64401dafeba" is_pre_election: true
I20260925 10:39:57.615319 30967 leader_election.cc:302] T ac5c65e3b8b44a7c96fa7e45f277c66a P e23f60bcefdf41b1843f6d5f5fcce1fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2971592598aa4a4b8f73d4450d8c92cc, e23f60bcefdf41b1843f6d5f5fcce1fb; no voters:
I20260925 10:39:57.615465 31182 raft_consensus.cc:2454] T ac5c65e3b8b44a7c96fa7e45f277c66a P 6bb6c395684244c384deb64401dafeba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e23f60bcefdf41b1843f6d5f5fcce1fb in term 0.
I20260925 10:39:57.615733 31471 raft_consensus.cc:2781] T ac5c65e3b8b44a7c96fa7e45f277c66a P e23f60bcefdf41b1843f6d5f5fcce1fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:39:57.615868 31471 raft_consensus.cc:482] T ac5c65e3b8b44a7c96fa7e45f277c66a P e23f60bcefdf41b1843f6d5f5fcce1fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:39:57.615983 31471 raft_consensus.cc:3037] T ac5c65e3b8b44a7c96fa7e45f277c66a P e23f60bcefdf41b1843f6d5f5fcce1fb [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:39:57.618067 31471 raft_consensus.cc:504] T ac5c65e3b8b44a7c96fa7e45f277c66a P e23f60bcefdf41b1843f6d5f5fcce1fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e23f60bcefdf41b1843f6d5f5fcce1fb" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 43887 } } peers { permanent_uuid: "6bb6c395684244c384deb64401dafeba" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 37033 } } peers { permanent_uuid: "2971592598aa4a4b8f73d4450d8c92cc" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 35055 } }
I20260925 10:39:57.618840 31471 leader_election.cc:288] T ac5c65e3b8b44a7c96fa7e45f277c66a P e23f60bcefdf41b1843f6d5f5fcce1fb [CANDIDATE]: Term 1 election: Requested vote from peers 6bb6c395684244c384deb64401dafeba (127.23.100.2:37033), 2971592598aa4a4b8f73d4450d8c92cc (127.23.100.3:35055)
I20260925 10:39:57.619174 31182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5c65e3b8b44a7c96fa7e45f277c66a" candidate_uuid: "e23f60bcefdf41b1843f6d5f5fcce1fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bb6c395684244c384deb64401dafeba"
I20260925 10:39:57.619422 31182 raft_consensus.cc:3037] T ac5c65e3b8b44a7c96fa7e45f277c66a P 6bb6c395684244c384deb64401dafeba [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:39:57.619463 31323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5c65e3b8b44a7c96fa7e45f277c66a" candidate_uuid: "e23f60bcefdf41b1843f6d5f5fcce1fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2971592598aa4a4b8f73d4450d8c92cc"
I20260925 10:39:57.619704 31323 raft_consensus.cc:3037] T ac5c65e3b8b44a7c96fa7e45f277c66a P 2971592598aa4a4b8f73d4450d8c92cc [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:39:57.621434 31182 raft_consensus.cc:2454] T ac5c65e3b8b44a7c96fa7e45f277c66a P 6bb6c395684244c384deb64401dafeba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e23f60bcefdf41b1843f6d5f5fcce1fb in term 1.
I20260925 10:39:57.621763 31323 raft_consensus.cc:2454] T ac5c65e3b8b44a7c96fa7e45f277c66a P 2971592598aa4a4b8f73d4450d8c92cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e23f60bcefdf41b1843f6d5f5fcce1fb in term 1.
I20260925 10:39:57.621796 30966 leader_election.cc:302] T ac5c65e3b8b44a7c96fa7e45f277c66a P e23f60bcefdf41b1843f6d5f5fcce1fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bb6c395684244c384deb64401dafeba, e23f60bcefdf41b1843f6d5f5fcce1fb; no voters:
I20260925 10:39:57.622097 31471 raft_consensus.cc:2781] T ac5c65e3b8b44a7c96fa7e45f277c66a P e23f60bcefdf41b1843f6d5f5fcce1fb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:39:57.622598 31471 raft_consensus.cc:686] T ac5c65e3b8b44a7c96fa7e45f277c66a P e23f60bcefdf41b1843f6d5f5fcce1fb [term 1 LEADER]: Becoming Leader. State: Replica: e23f60bcefdf41b1843f6d5f5fcce1fb, State: Running, Role: LEADER
I20260925 10:39:57.623020 31471 consensus_queue.cc:237] T ac5c65e3b8b44a7c96fa7e45f277c66a P e23f60bcefdf41b1843f6d5f5fcce1fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e23f60bcefdf41b1843f6d5f5fcce1fb" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 43887 } } peers { permanent_uuid: "6bb6c395684244c384deb64401dafeba" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 37033 } } peers { permanent_uuid: "2971592598aa4a4b8f73d4450d8c92cc" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 35055 } }
I20260925 10:39:57.623363 31080 heartbeater.cc:503] Master 127.23.100.61:38343 was elected leader, sending a full tablet report...
I20260925 10:39:57.623557 31473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:57.623134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:33367 (local address 127.23.100.1:47369)
0925 10:39:57.623271 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:57.623415 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:33367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 10:39:57.626688 30710 catalog_manager.cc:5495] T ac5c65e3b8b44a7c96fa7e45f277c66a P e23f60bcefdf41b1843f6d5f5fcce1fb reported cstate change: term changed from 0 to 1, leader changed from <none> to e23f60bcefdf41b1843f6d5f5fcce1fb (127.23.100.1). New cstate: current_term: 1 leader_uuid: "e23f60bcefdf41b1843f6d5f5fcce1fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e23f60bcefdf41b1843f6d5f5fcce1fb" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 43887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6bb6c395684244c384deb64401dafeba" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 37033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2971592598aa4a4b8f73d4450d8c92cc" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 35055 } health_report { overall_health: UNKNOWN } } }
I20260925 10:39:57.650694 31457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:57.644245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46582 (local address 127.23.100.1:43887)
0925 10:39:57.644434 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:57.644439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:57.644455 (+ 16us) server_negotiation.cc:407] Connection header received
0925 10:39:57.644665 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:57.644668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:57.644752 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:57.644887 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:39:57.645272 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:57.646042 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:57.648315 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:57.648789 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:57.648898 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:57.649443 (+ 545us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:39:57.650028 (+ 585us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:39:57.650152 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:57.650289 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 10:39:57.650425 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 10:39:57.655301 31182 raft_consensus.cc:1261] T ac5c65e3b8b44a7c96fa7e45f277c66a P 6bb6c395684244c384deb64401dafeba [term 1 FOLLOWER]: Refusing update from remote peer e23f60bcefdf41b1843f6d5f5fcce1fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:39:57.655385 31323 raft_consensus.cc:1261] T ac5c65e3b8b44a7c96fa7e45f277c66a P 2971592598aa4a4b8f73d4450d8c92cc [term 1 FOLLOWER]: Refusing update from remote peer e23f60bcefdf41b1843f6d5f5fcce1fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:39:57.656131 31474 consensus_queue.cc:1035] T ac5c65e3b8b44a7c96fa7e45f277c66a P e23f60bcefdf41b1843f6d5f5fcce1fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bb6c395684244c384deb64401dafeba" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 37033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:39:57.656527 31471 consensus_queue.cc:1035] T ac5c65e3b8b44a7c96fa7e45f277c66a P e23f60bcefdf41b1843f6d5f5fcce1fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "2971592598aa4a4b8f73d4450d8c92cc" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 35055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:39:57.659337 31469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:57.658910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:33367 (local address 127.23.100.2:58049)
0925 10:39:57.659112 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:57.659188 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:33367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260925 10:39:57.659473 31466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:57.658832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:33367 (local address 127.23.100.3:56127)
0925 10:39:57.659126 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:57.659269 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:33367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
I20260925 10:39:57.664363 31477 mvcc.cc:205] Tried to move back new op lower bound from 7333203139188076544 to 7333203139068002304. Current Snapshot: MvccSnapshot[applied={T|T < 7333203139188076544}]
I20260925 10:39:57.666386 31466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:57.665737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:33367 (local address 127.23.100.3:50119)
0925 10:39:57.666038 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:57.666148 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:33367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
I20260925 10:39:57.667346 31473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:57.666379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:33367 (local address 127.23.100.1:39327)
0925 10:39:57.667098 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:57.667181 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:33367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":613}
I20260925 10:39:57.679541 31473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:57.679054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:33367 (local address 127.23.100.1:43319)
0925 10:39:57.679254 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:57.679378 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:33367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260925 10:39:57.681572 31469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:57.680876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:33367 (local address 127.23.100.2:53535)
0925 10:39:57.681333 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:57.681425 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:33367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352}
I20260925 10:39:57.682871 30710 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:39:57.683120 30710 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:39:57.688180 30710 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d1dd4804029c47089cbf66ab6106c8c5: Sending DeleteTablet for 3 replicas of tablet ac5c65e3b8b44a7c96fa7e45f277c66a
I20260925 10:39:57.689054 31013 tablet_service.cc:1514] Processing DeleteTablet for tablet ac5c65e3b8b44a7c96fa7e45f277c66a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:39:57 UTC) from {username='oryx', principal='oryx/127.23.100.61@KRBTEST.COM'} at 127.0.0.1:46566
I20260925 10:39:57.689431 31303 tablet_service.cc:1514] Processing DeleteTablet for tablet ac5c65e3b8b44a7c96fa7e45f277c66a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:39:57 UTC) from {username='oryx', principal='oryx/127.23.100.61@KRBTEST.COM'} at 127.0.0.1:41136
I20260925 10:39:57.689735 31487 tablet_replica.cc:330] T ac5c65e3b8b44a7c96fa7e45f277c66a P e23f60bcefdf41b1843f6d5f5fcce1fb: stopping tablet replica
I20260925 10:39:57.689909 31487 raft_consensus.cc:2229] T ac5c65e3b8b44a7c96fa7e45f277c66a P e23f60bcefdf41b1843f6d5f5fcce1fb [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:39:57.690038 31162 tablet_service.cc:1514] Processing DeleteTablet for tablet ac5c65e3b8b44a7c96fa7e45f277c66a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:39:57 UTC) from {username='oryx', principal='oryx/127.23.100.61@KRBTEST.COM'} at 127.0.0.1:41510
I20260925 10:39:57.690505 31487 raft_consensus.cc:2258] T ac5c65e3b8b44a7c96fa7e45f277c66a P e23f60bcefdf41b1843f6d5f5fcce1fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:39:57.691205 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 30898
I20260925 10:39:57.694458 31487 ts_tablet_manager.cc:1905] T ac5c65e3b8b44a7c96fa7e45f277c66a P e23f60bcefdf41b1843f6d5f5fcce1fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:39:57.700922 31489 tablet_replica.cc:330] T ac5c65e3b8b44a7c96fa7e45f277c66a P 6bb6c395684244c384deb64401dafeba: stopping tablet replica
I20260925 10:39:57.701177 31489 raft_consensus.cc:2229] T ac5c65e3b8b44a7c96fa7e45f277c66a P 6bb6c395684244c384deb64401dafeba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:39:57.701547 31489 raft_consensus.cc:2258] T ac5c65e3b8b44a7c96fa7e45f277c66a P 6bb6c395684244c384deb64401dafeba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:39:57.702663 31489 ts_tablet_manager.cc:1905] T ac5c65e3b8b44a7c96fa7e45f277c66a P 6bb6c395684244c384deb64401dafeba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:39:57.705629 31487 ts_tablet_manager.cc:1918] T ac5c65e3b8b44a7c96fa7e45f277c66a P e23f60bcefdf41b1843f6d5f5fcce1fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:39:57.705790 31487 log.cc:1196] T ac5c65e3b8b44a7c96fa7e45f277c66a P e23f60bcefdf41b1843f6d5f5fcce1fb: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-0/wal/wals/ac5c65e3b8b44a7c96fa7e45f277c66a
I20260925 10:39:57.706328 31487 ts_tablet_manager.cc:1939] T ac5c65e3b8b44a7c96fa7e45f277c66a P e23f60bcefdf41b1843f6d5f5fcce1fb: Deleting consensus metadata
I20260925 10:39:57.707522 30698 catalog_manager.cc:4841] TS e23f60bcefdf41b1843f6d5f5fcce1fb (127.23.100.1:43887): tablet ac5c65e3b8b44a7c96fa7e45f277c66a (table test-table [id=11ad5bca444b405b8bc1fac854638450]) successfully deleted
I20260925 10:39:57.709208 31488 tablet_replica.cc:330] T ac5c65e3b8b44a7c96fa7e45f277c66a P 2971592598aa4a4b8f73d4450d8c92cc: stopping tablet replica
I20260925 10:39:57.709398 31488 raft_consensus.cc:2229] T ac5c65e3b8b44a7c96fa7e45f277c66a P 2971592598aa4a4b8f73d4450d8c92cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:39:57.709746 31489 ts_tablet_manager.cc:1918] T ac5c65e3b8b44a7c96fa7e45f277c66a P 6bb6c395684244c384deb64401dafeba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:39:57.709801 31488 raft_consensus.cc:2258] T ac5c65e3b8b44a7c96fa7e45f277c66a P 2971592598aa4a4b8f73d4450d8c92cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:39:57.709887 31489 log.cc:1196] T ac5c65e3b8b44a7c96fa7e45f277c66a P 6bb6c395684244c384deb64401dafeba: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-1/wal/wals/ac5c65e3b8b44a7c96fa7e45f277c66a
I20260925 10:39:57.710357 31489 ts_tablet_manager.cc:1939] T ac5c65e3b8b44a7c96fa7e45f277c66a P 6bb6c395684244c384deb64401dafeba: Deleting consensus metadata
I20260925 10:39:57.711488 30696 catalog_manager.cc:4841] TS 6bb6c395684244c384deb64401dafeba (127.23.100.2:37033): tablet ac5c65e3b8b44a7c96fa7e45f277c66a (table test-table [id=11ad5bca444b405b8bc1fac854638450]) successfully deleted
I20260925 10:39:57.716675 31457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:57.692183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46598 (local address 127.23.100.1:43887)
0925 10:39:57.692333 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:57.692338 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:57.701577 (+ 9239us) server_negotiation.cc:407] Connection header received
0925 10:39:57.701975 (+ 398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:57.701980 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:57.702109 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:57.702214 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:57.703633 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:57.704767 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:57.706841 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:57.707480 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:57.707583 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:57.710798 (+ 3215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:57.710809 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:57.710822 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:57.710858 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:57.713898 (+ 3040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:57.714663 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:57.714670 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:57.714672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:57.714814 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:57.715201 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:57.715206 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:57.715215 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:57.715541 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:57.715836 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:57.716128 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 10:39:57.716322 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 10:39:57.717437 31073 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:39:57.717509 31488 ts_tablet_manager.cc:1905] T ac5c65e3b8b44a7c96fa7e45f277c66a P 2971592598aa4a4b8f73d4450d8c92cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:39:57.723373 31488 ts_tablet_manager.cc:1918] T ac5c65e3b8b44a7c96fa7e45f277c66a P 2971592598aa4a4b8f73d4450d8c92cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:39:57.723517 31488 log.cc:1196] T ac5c65e3b8b44a7c96fa7e45f277c66a P 2971592598aa4a4b8f73d4450d8c92cc: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790332740062128-23952-0/minicluster-data/ts-2/wal/wals/ac5c65e3b8b44a7c96fa7e45f277c66a
I20260925 10:39:57.723897 31488 ts_tablet_manager.cc:1939] T ac5c65e3b8b44a7c96fa7e45f277c66a P 2971592598aa4a4b8f73d4450d8c92cc: Deleting consensus metadata
I20260925 10:39:57.724932 30697 catalog_manager.cc:4841] TS 2971592598aa4a4b8f73d4450d8c92cc (127.23.100.3:35055): tablet ac5c65e3b8b44a7c96fa7e45f277c66a (table test-table [id=11ad5bca444b405b8bc1fac854638450]) successfully deleted
I20260925 10:39:57.818027 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 30898
I20260925 10:39:57.838922 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 31090
I20260925 10:39:57.852325 31460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:57.839751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41516 (local address 127.23.100.2:37033)
0925 10:39:57.839910 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:57.839917 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:57.839985 (+ 68us) server_negotiation.cc:407] Connection header received
0925 10:39:57.840211 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:57.840215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:57.840276 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:57.840382 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:57.841767 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:57.842626 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:57.843821 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:57.844216 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:57.844418 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:57.846881 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:57.846905 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:57.846924 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:57.846980 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:57.849543 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:57.850219 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:57.850229 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:57.850234 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:57.850452 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:57.850805 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:57.850813 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:57.850818 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:57.851170 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:57.851477 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:57.851755 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 10:39:57.851970 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 10:39:57.853019 31222 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:39:57.944511 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 31090
I20260925 10:39:57.961970 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 31240
I20260925 10:39:57.974607 31458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:57.962697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41150 (local address 127.23.100.3:35055)
0925 10:39:57.962885 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 10:39:57.962890 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:39:57.962911 (+ 21us) server_negotiation.cc:407] Connection header received
0925 10:39:57.963056 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:39:57.963060 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:39:57.963120 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:39:57.963201 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:39:57.964460 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:57.965139 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:57.966464 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:39:57.966853 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:39:57.966946 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:39:57.969342 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:39:57.969357 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:39:57.969371 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:39:57.969401 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:39:57.972244 (+ 2843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:57.972838 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:57.972849 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:57.972856 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:57.972998 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:39:57.973306 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:39:57.973309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:39:57.973311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:39:57.973646 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:39:57.973861 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0925 10:39:57.974044 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 10:39:57.974189 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 10:39:57.975296 31363 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:39:58.077704 31494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:58.077001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:33367 (local address 127.0.0.1:42722)
0925 10:39:58.077453 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:58.077539 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:33367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":260,"threads_started":1}
I20260925 10:39:58.079039 31494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:39:58.078723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:33367 (local address 127.0.0.1:42726)
0925 10:39:58.078866 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 10:39:58.078933 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:33367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 10:39:58.094033 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 31240
I20260925 10:39:58.115540 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 30576
I20260925 10:39:58.116479 30742 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:39:58.238350 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 30576
W20260925 10:39:58.258484 30885 connection.cc:535] server connection from 127.0.0.1:37750 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 10:39:58.258863 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 30676
I20260925 10:39:58.259804 30929 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:39:58.375630 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 30676
2026-09-25T10:39:58Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9113 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:39:58 dist-test-slave-dxdd krb5kdc[31500](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:39:58 dist-test-slave-dxdd krb5kdc[31500](info): set up 2 sockets
Sep 25 10:39:58 dist-test-slave-dxdd krb5kdc[31500](info): commencing operation
krb5kdc: starting...
W20260925 10:39:59.459334 23952 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.001s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Sep 25 10:39:59 dist-test-slave-dxdd krb5kdc[31500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:39:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:39:59Z Disabled control of system clock
WARNING: no policy specified for oryx/127.23.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/krb5kdc/oryx_127.23.100.62.keytab.
Entry for principal oryx/127.23.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/krb5kdc/oryx_127.23.100.62.keytab.
I20260925 10:39:59.657758 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:45071
--webserver_interface=127.23.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/krb5kdc/oryx_127.23.100.62.keytab
--principal=oryx/127.23.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:45727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.62:45071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:39:59.968230 31513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:39:59.968483 31513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:39:59.968648 31513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:39:59.975546 31513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:39:59.975656 31513 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 10:39:59.975723 31513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:39:59.975831 31513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:39:59.975911 31513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:39:59.976011 31513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:39:59.985427 31513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:45727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.62:45071
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/krb5kdc/oryx_127.23.100.62.keytab
--principal=oryx/127.23.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:45071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:39:59.986670 31513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:39:59.987938 31513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:39:59.999225 31521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:59.999297 31519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:39:59.999600 31518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:40:00.000397 31513 server_base.cc:1034] running on GCE node
I20260925 10:40:00.001142 31513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:40:00.002629 31513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:40:00.003875 31513 hybrid_clock.cc:648] HybridClock initialized: now 1790332800003802 us; error 50 us; skew 500 ppm
I20260925 10:40:00.004182 31513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:40:00 dist-test-slave-dxdd krb5kdc[31500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332800, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:40:00.007318 31513 init.cc:377] Logged in from keytab as oryx/127.23.100.62@KRBTEST.COM (short username oryx)
I20260925 10:40:00.008633 31513 webserver.cc:460] Webserver started at http://127.23.100.62:43781/ using document root <none> and password file <none>
I20260925 10:40:00.009287 31513 fs_manager.cc:359] Metadata directory not provided
I20260925 10:40:00.009433 31513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:40:00.009738 31513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:40:00.012626 31513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/master-0/data/instance:
uuid: "2620a3124bff4655ac88451061702e7d"
format_stamp: "Formatted at 2026-09-25 10:40:00 on dist-test-slave-dxdd"
server_key: "60ebe9e9c1a01d5d278a3532523e64cb"
server_key_iv: "63085883d9fe592169a4c02cfd276039"
server_key_version: "encryptionkey@0"
I20260925 10:40:00.013409 31513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/master-0/wal/instance:
uuid: "2620a3124bff4655ac88451061702e7d"
format_stamp: "Formatted at 2026-09-25 10:40:00 on dist-test-slave-dxdd"
server_key: "60ebe9e9c1a01d5d278a3532523e64cb"
server_key_iv: "63085883d9fe592169a4c02cfd276039"
server_key_version: "encryptionkey@0"
I20260925 10:40:00.018365 31513 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 10:40:00.021641 31528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:40:00.022869 31513 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 10:40:00.023039 31513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/master-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/master-0/wal
uuid: "2620a3124bff4655ac88451061702e7d"
format_stamp: "Formatted at 2026-09-25 10:40:00 on dist-test-slave-dxdd"
server_key: "60ebe9e9c1a01d5d278a3532523e64cb"
server_key_iv: "63085883d9fe592169a4c02cfd276039"
server_key_version: "encryptionkey@0"
I20260925 10:40:00.023239 31513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:40:00.045445 31513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:40:00.049803 31513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:40:00.050073 31513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:40:00.070250 31513 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.62:45071
I20260925 10:40:00.070284 31589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.62:45071 every 8 connection(s)
I20260925 10:40:00.071523 31513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
I20260925 10:40:00.074997 31590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:40:00.076874 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 31513
I20260925 10:40:00.077076 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/master-0/wal/instance
I20260925 10:40:00.077685 23952 external_mini_cluster.cc:1428] Setting key 4ac1c3c3eb8a37770da01f1878144ee1
I20260925 10:40:00.087129 31590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2620a3124bff4655ac88451061702e7d: Bootstrap starting.
I20260925 10:40:00.090500 31590 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2620a3124bff4655ac88451061702e7d: Neither blocks nor log segments found. Creating new log.
I20260925 10:40:00.091394 31590 log.cc:824] T 00000000000000000000000000000000 P 2620a3124bff4655ac88451061702e7d: Log is configured to *not* fsync() on all Append() calls
Sep 25 10:40:00 dist-test-slave-dxdd krb5kdc[31500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM
I20260925 10:40:00.094184 31590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2620a3124bff4655ac88451061702e7d: No bootstrap required, opened a new log
I20260925 10:40:00.101627 31593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:00.079650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51064 (local address 127.23.100.62:45071)
0925 10:40:00.080496 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:00.080511 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:00.080560 (+ 49us) server_negotiation.cc:407] Connection header received
0925 10:40:00.082837 (+ 2277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:00.082862 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:00.083338 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:00.083750 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:00.084904 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:00.086815 (+ 1911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:00.088123 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:00.089214 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:00.089383 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:00.093497 (+ 4114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:40:00.093504 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:40:00.093528 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:40:00.093590 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:40:00.097404 (+ 3814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:40:00.098020 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:40:00.098028 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:40:00.098036 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:40:00.098230 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:40:00.098632 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:40:00.098638 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:40:00.098642 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:40:00.099251 (+ 609us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:40:00.099614 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:00.099915 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 10:40:00.100322 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":361,"threads_started":1}
I20260925 10:40:00.104107 31590 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2620a3124bff4655ac88451061702e7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2620a3124bff4655ac88451061702e7d" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 45071 } }
I20260925 10:40:00.104512 31590 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2620a3124bff4655ac88451061702e7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:40:00.104594 31590 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2620a3124bff4655ac88451061702e7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2620a3124bff4655ac88451061702e7d, State: Initialized, Role: FOLLOWER
I20260925 10:40:00.105039 31590 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2620a3124bff4655ac88451061702e7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2620a3124bff4655ac88451061702e7d" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 45071 } }
I20260925 10:40:00.105233 31590 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2620a3124bff4655ac88451061702e7d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:40:00.105324 31590 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2620a3124bff4655ac88451061702e7d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:40:00.105440 31590 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2620a3124bff4655ac88451061702e7d [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:40:00.107504 31590 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2620a3124bff4655ac88451061702e7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2620a3124bff4655ac88451061702e7d" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 45071 } }
I20260925 10:40:00.108047 31590 leader_election.cc:302] T 00000000000000000000000000000000 P 2620a3124bff4655ac88451061702e7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2620a3124bff4655ac88451061702e7d; no voters:
I20260925 10:40:00.108672 31590 leader_election.cc:288] T 00000000000000000000000000000000 P 2620a3124bff4655ac88451061702e7d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:40:00.108748 31595 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2620a3124bff4655ac88451061702e7d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:40:00.109522 31595 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2620a3124bff4655ac88451061702e7d [term 1 LEADER]: Becoming Leader. State: Replica: 2620a3124bff4655ac88451061702e7d, State: Running, Role: LEADER
I20260925 10:40:00.110031 31595 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2620a3124bff4655ac88451061702e7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2620a3124bff4655ac88451061702e7d" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 45071 } }
I20260925 10:40:00.110682 31590 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2620a3124bff4655ac88451061702e7d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:40:00.112807 31597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2620a3124bff4655ac88451061702e7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2620a3124bff4655ac88451061702e7d. Latest consensus state: current_term: 1 leader_uuid: "2620a3124bff4655ac88451061702e7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2620a3124bff4655ac88451061702e7d" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 45071 } } }
I20260925 10:40:00.112730 31596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2620a3124bff4655ac88451061702e7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2620a3124bff4655ac88451061702e7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2620a3124bff4655ac88451061702e7d" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 45071 } } }
I20260925 10:40:00.113161 31597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2620a3124bff4655ac88451061702e7d [sys.catalog]: This master's current role is: LEADER
I20260925 10:40:00.113173 31596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2620a3124bff4655ac88451061702e7d [sys.catalog]: This master's current role is: LEADER
I20260925 10:40:00.115171 31603 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:40:00.120173 31603 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:40:00.128110 31603 catalog_manager.cc:1347] Generated new cluster ID: 130cb355ec0441669b44cad4b261a8d3
I20260925 10:40:00.128203 31603 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:40:00.149711 31603 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:40:00.151063 31603 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:40:00.156411 31603 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2620a3124bff4655ac88451061702e7d: Generated new TSK 0
I20260925 10:40:00.157266 31603 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.23.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/krb5kdc/oryx_127.23.100.1.keytab.
Entry for principal oryx/127.23.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/krb5kdc/oryx_127.23.100.1.keytab.
I20260925 10:40:00.218237 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.1:0
--local_ip_for_outbound_sockets=127.23.100.1
--webserver_interface=127.23.100.1
--webserver_port=0
--tserver_master_addrs=127.23.100.62:45071
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/krb5kdc/oryx_127.23.100.1.keytab
--principal=oryx/127.23.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:45727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:40:00.553764 31616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:40:00.554008 31616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:40:00.554100 31616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:40:00.561094 31616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:40:00.561189 31616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:40:00.561283 31616 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 10:40:00.561389 31616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.1
I20260925 10:40:00.571188 31616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:45727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/krb5kdc/oryx_127.23.100.1.keytab
--principal=oryx/127.23.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.100.62:45071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.1
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:40:00.572234 31616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:40:00.573458 31616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:40:00.586035 31621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:40:00.586062 31622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:40:00.586231 31624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:40:00.586830 31616 server_base.cc:1034] running on GCE node
I20260925 10:40:00.587349 31616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:40:00.588183 31616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:40:00.589380 31616 hybrid_clock.cc:648] HybridClock initialized: now 1790332800589326 us; error 30 us; skew 500 ppm
I20260925 10:40:00.589674 31616 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:40:00 dist-test-slave-dxdd krb5kdc[31500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332800, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:40:00.592684 31616 init.cc:377] Logged in from keytab as oryx/127.23.100.1@KRBTEST.COM (short username oryx)
I20260925 10:40:00.594013 31616 webserver.cc:460] Webserver started at http://127.23.100.1:45663/ using document root <none> and password file <none>
I20260925 10:40:00.594866 31616 fs_manager.cc:359] Metadata directory not provided
I20260925 10:40:00.594992 31616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:40:00.595306 31616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:40:00.598026 31616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-0/data/instance:
uuid: "dcafa1392bf049e4b508fb1ef2a74889"
format_stamp: "Formatted at 2026-09-25 10:40:00 on dist-test-slave-dxdd"
server_key: "3a9d50f364351ddb1aed04ca8c0b85cb"
server_key_iv: "fcfc69221b788b862fbcd6619df4eb84"
server_key_version: "encryptionkey@0"
I20260925 10:40:00.598928 31616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-0/wal/instance:
uuid: "dcafa1392bf049e4b508fb1ef2a74889"
format_stamp: "Formatted at 2026-09-25 10:40:00 on dist-test-slave-dxdd"
server_key: "3a9d50f364351ddb1aed04ca8c0b85cb"
server_key_iv: "fcfc69221b788b862fbcd6619df4eb84"
server_key_version: "encryptionkey@0"
I20260925 10:40:00.603976 31616 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 10:40:00.607223 31631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:40:00.608493 31616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 10:40:00.608740 31616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-0/wal
uuid: "dcafa1392bf049e4b508fb1ef2a74889"
format_stamp: "Formatted at 2026-09-25 10:40:00 on dist-test-slave-dxdd"
server_key: "3a9d50f364351ddb1aed04ca8c0b85cb"
server_key_iv: "fcfc69221b788b862fbcd6619df4eb84"
server_key_version: "encryptionkey@0"
I20260925 10:40:00.609038 31616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:40:00.638760 31616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:40:00.642659 31616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:40:00.642925 31616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:40:00.645391 31616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:40:00.645495 31616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:40:00.645660 31616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:40:00.645740 31616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 10:40:00 dist-test-slave-dxdd krb5kdc[31500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332800, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.1@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM
I20260925 10:40:00.670428 31640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:00.646110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:46105 (local address 127.23.100.62:45071)
0925 10:40:00.646659 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:00.646666 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:00.647796 (+ 1130us) server_negotiation.cc:407] Connection header received
0925 10:40:00.649765 (+ 1969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:00.649778 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:00.649867 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:00.650001 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:00.652651 (+ 2650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:00.653588 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:00.656686 (+ 3098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:00.657305 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:00.657443 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:00.662759 (+ 5316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:40:00.662769 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:40:00.662776 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:40:00.662823 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:40:00.666643 (+ 3820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:40:00.667816 (+ 1173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:40:00.667823 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:40:00.667829 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:40:00.667988 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:40:00.668485 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:40:00.668490 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:40:00.668494 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:40:00.668764 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:40:00.668938 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:00.669814 (+ 876us) server_negotiation.cc:299] Negotiation successful
0925 10:40:00.670026 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":321,"threads_started":1}
I20260925 10:40:00.671485 31641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:00.646763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:45071 (local address 127.23.100.1:46105)
0925 10:40:00.647609 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:00.647663 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:00.649451 (+ 1788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:00.650471 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:00.650488 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:00.651439 (+ 951us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:00.652439 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:00.652469 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:00.653785 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:00.653793 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:00.656523 (+ 2730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:00.656546 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:00.657620 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:00.657627 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:00.657978 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:00.658792 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:00.658827 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:00.662481 (+ 3654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:40:00.666839 (+ 4358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:40:00.666851 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:40:00.666875 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:40:00.667603 (+ 728us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:40:00.668103 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:40:00.668110 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:40:00.668116 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:40:00.668348 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:40:00.669024 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:40:00.669034 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:40:00.669430 (+ 396us) client_negotiation.cc:769] Sending connection context
0925 10:40:00.669720 (+ 290us) client_negotiation.cc:240] Negotiation successful
0925 10:40:00.670219 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":391,"threads_started":1}
I20260925 10:40:00.691458 31718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:00.678114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:45071 (local address 127.23.100.1:33029)
0925 10:40:00.678853 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:00.678890 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:00.679054 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:00.679362 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:00.679370 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:00.680092 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:00.680555 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:00.680571 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:00.681522 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:00.681530 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:00.683624 (+ 2094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:00.683643 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:00.684577 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:00.684595 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:00.684850 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:00.685499 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:00.685514 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:00.686250 (+ 736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:40:00.689096 (+ 2846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:40:00.689101 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:40:00.689105 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:40:00.689443 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:40:00.689807 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:40:00.689812 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:40:00.689815 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:40:00.689937 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:40:00.690454 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:40:00.690458 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:40:00.690680 (+ 222us) client_negotiation.cc:769] Sending connection context
0925 10:40:00.690864 (+ 184us) client_negotiation.cc:240] Negotiation successful
0925 10:40:00.691164 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":382,"threads_started":1}
I20260925 10:40:00.691817 31640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:00.678128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:33029 (local address 127.23.100.62:45071)
0925 10:40:00.678364 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:00.678372 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:00.678989 (+ 617us) server_negotiation.cc:407] Connection header received
0925 10:40:00.679168 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:00.679172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:00.679227 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:00.679301 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:00.680700 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:00.681372 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:00.683841 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:00.684356 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:00.684483 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:00.686507 (+ 2024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:40:00.686517 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:40:00.686520 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:40:00.686548 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:40:00.688978 (+ 2430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:40:00.689572 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:40:00.689581 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:40:00.689583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:40:00.689695 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:40:00.690044 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:40:00.690048 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:40:00.690051 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:40:00.690297 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:40:00.690485 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:00.691030 (+ 545us) server_negotiation.cc:299] Negotiation successful
0925 10:40:00.691332 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 10:40:00.700328 31616 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.1:37725
I20260925 10:40:00.700369 31751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.1:37725 every 8 connection(s)
I20260925 10:40:00.701506 31616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
I20260925 10:40:00.702739 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 31616
I20260925 10:40:00.702941 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-0/wal/instance
I20260925 10:40:00.703603 31752 heartbeater.cc:347] Connected to a master server at 127.23.100.62:45071
I20260925 10:40:00.703778 23952 external_mini_cluster.cc:1428] Setting key 10b77ad94e1f37f130c72ee0a621afe1
I20260925 10:40:00.703832 31752 heartbeater.cc:464] Registering TS with master...
I20260925 10:40:00.704548 31752 heartbeater.cc:511] Master 127.23.100.62:45071 requested a full tablet report, sending...
I20260925 10:40:00.706645 31545 ts_manager.cc:194] Registered new tserver with Master: dcafa1392bf049e4b508fb1ef2a74889 (127.23.100.1:37725)
I20260925 10:40:00.708014 31545 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.100.1@KRBTEST.COM'} at 127.23.100.1:46105
WARNING: no policy specified for oryx/127.23.100.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.100.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/krb5kdc/oryx_127.23.100.2.keytab.
Entry for principal oryx/127.23.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/krb5kdc/oryx_127.23.100.2.keytab.
I20260925 10:40:00.749198 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.2:0
--local_ip_for_outbound_sockets=127.23.100.2
--webserver_interface=127.23.100.2
--webserver_port=0
--tserver_master_addrs=127.23.100.62:45071
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/krb5kdc/oryx_127.23.100.2.keytab
--principal=oryx/127.23.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:45727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:40:01.060197 31756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:40:01.060482 31756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:40:01.060648 31756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:40:01.067517 31756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:40:01.067651 31756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:40:01.067785 31756 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 10:40:01.067919 31756 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.2
I20260925 10:40:01.081765 31756 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:45727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/krb5kdc/oryx_127.23.100.2.keytab
--principal=oryx/127.23.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.100.62:45071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.2
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:40:01.083006 31756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:40:01.084259 31756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:40:01.096977 31761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:40:01.097069 31764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:40:01.097047 31762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:40:01.097550 31756 server_base.cc:1034] running on GCE node
I20260925 10:40:01.098086 31756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:40:01.098984 31756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:40:01.100219 31756 hybrid_clock.cc:648] HybridClock initialized: now 1790332801100146 us; error 51 us; skew 500 ppm
I20260925 10:40:01.100495 31756 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:40:01 dist-test-slave-dxdd krb5kdc[31500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332801, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:40:01.103549 31756 init.cc:377] Logged in from keytab as oryx/127.23.100.2@KRBTEST.COM (short username oryx)
I20260925 10:40:01.104846 31756 webserver.cc:460] Webserver started at http://127.23.100.2:38321/ using document root <none> and password file <none>
I20260925 10:40:01.105474 31756 fs_manager.cc:359] Metadata directory not provided
I20260925 10:40:01.105598 31756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:40:01.105906 31756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:40:01.108677 31756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-1/data/instance:
uuid: "d78e073456484d9082e78231eee41e48"
format_stamp: "Formatted at 2026-09-25 10:40:01 on dist-test-slave-dxdd"
server_key: "3b361c0cadd0b828067452fc7146b73e"
server_key_iv: "5f7075c40b57f993cb25443a7119a6c1"
server_key_version: "encryptionkey@0"
I20260925 10:40:01.109416 31756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-1/wal/instance:
uuid: "d78e073456484d9082e78231eee41e48"
format_stamp: "Formatted at 2026-09-25 10:40:01 on dist-test-slave-dxdd"
server_key: "3b361c0cadd0b828067452fc7146b73e"
server_key_iv: "5f7075c40b57f993cb25443a7119a6c1"
server_key_version: "encryptionkey@0"
I20260925 10:40:01.114519 31756 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260925 10:40:01.117795 31771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:40:01.119019 31756 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:40:01.119171 31756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-1/wal
uuid: "d78e073456484d9082e78231eee41e48"
format_stamp: "Formatted at 2026-09-25 10:40:01 on dist-test-slave-dxdd"
server_key: "3b361c0cadd0b828067452fc7146b73e"
server_key_iv: "5f7075c40b57f993cb25443a7119a6c1"
server_key_version: "encryptionkey@0"
I20260925 10:40:01.119369 31756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:40:01.136642 31756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:40:01.140323 31756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:40:01.140575 31756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:40:01.142992 31756 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:40:01.143096 31756 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:40:01.143285 31756 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:40:01.143353 31756 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 10:40:01 dist-test-slave-dxdd krb5kdc[31500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332801, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.2@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM
I20260925 10:40:01.169299 31780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:01.143930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:45071 (local address 127.23.100.2:36821)
0925 10:40:01.144809 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:01.144851 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:01.146905 (+ 2054us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:01.147917 (+ 1012us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:01.147931 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:01.148913 (+ 982us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:01.149836 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:01.149869 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:01.151355 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:01.151363 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:01.154179 (+ 2816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:01.154198 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:01.155251 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:01.155260 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:01.155635 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:01.156379 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:01.156418 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:01.160066 (+ 3648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:40:01.163836 (+ 3770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:40:01.163845 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:40:01.163859 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:40:01.164662 (+ 803us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:40:01.165121 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:40:01.165151 (+ 30us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:40:01.165158 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:40:01.165469 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:40:01.166158 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:40:01.166165 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:40:01.166699 (+ 534us) client_negotiation.cc:769] Sending connection context
0925 10:40:01.167017 (+ 318us) client_negotiation.cc:240] Negotiation successful
0925 10:40:01.167636 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":410,"threads_started":1}
I20260925 10:40:01.170706 31640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:01.143401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:36821 (local address 127.23.100.62:45071)
0925 10:40:01.143634 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:01.143640 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:01.144991 (+ 1351us) server_negotiation.cc:407] Connection header received
0925 10:40:01.147230 (+ 2239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:01.147236 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:01.147325 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:01.147438 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:01.150117 (+ 2679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:01.151176 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:01.154371 (+ 3195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:01.154967 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:01.155110 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:01.160277 (+ 5167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:40:01.160284 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:40:01.160289 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:40:01.160332 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:40:01.163694 (+ 3362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:40:01.164834 (+ 1140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:40:01.164842 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:40:01.164847 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:40:01.164996 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:40:01.165597 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:40:01.165603 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:40:01.165606 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:40:01.166037 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:40:01.166180 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:01.170082 (+ 3902us) server_negotiation.cc:299] Negotiation successful
0925 10:40:01.170370 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260925 10:40:01.189291 31756 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.2:35365
I20260925 10:40:01.189322 31890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.2:35365 every 8 connection(s)
I20260925 10:40:01.190734 31756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
I20260925 10:40:01.193041 31891 heartbeater.cc:347] Connected to a master server at 127.23.100.62:45071
I20260925 10:40:01.193262 31891 heartbeater.cc:464] Registering TS with master...
I20260925 10:40:01.194079 31891 heartbeater.cc:511] Master 127.23.100.62:45071 requested a full tablet report, sending...
I20260925 10:40:01.195674 31545 ts_manager.cc:194] Registered new tserver with Master: d78e073456484d9082e78231eee41e48 (127.23.100.2:35365)
I20260925 10:40:01.196285 31640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:01.180816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:45365 (local address 127.23.100.62:45071)
0925 10:40:01.181001 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:01.181006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:01.181887 (+ 881us) server_negotiation.cc:407] Connection header received
0925 10:40:01.182107 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:01.182113 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:01.182187 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:01.182288 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:01.183765 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:01.184477 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:01.187290 (+ 2813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:01.187775 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:01.187869 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:01.190220 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:40:01.190224 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:40:01.190227 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:40:01.190254 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:40:01.193536 (+ 3282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:40:01.194350 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:40:01.194355 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:40:01.194359 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:40:01.194486 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:40:01.194871 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:40:01.194875 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:40:01.194877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:40:01.195278 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:40:01.195427 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:01.195905 (+ 478us) server_negotiation.cc:299] Negotiation successful
0925 10:40:01.196076 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 10:40:01.196476 31545 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.100.2@KRBTEST.COM'} at 127.23.100.2:36821
I20260925 10:40:01.197000 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 31756
I20260925 10:40:01.197172 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-1/wal/instance
I20260925 10:40:01.197185 31870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:01.180973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:45071 (local address 127.23.100.2:45365)
0925 10:40:01.181768 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:01.181787 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:01.181952 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:01.182331 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:01.182336 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:01.182964 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:01.183572 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:01.183596 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:01.184733 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:01.184739 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:01.187109 (+ 2370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:01.187131 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:01.187999 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:01.188009 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:01.188207 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:01.188984 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:01.189011 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:01.190057 (+ 1046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:40:01.193659 (+ 3602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:40:01.193664 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:40:01.193667 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:40:01.194201 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:40:01.194595 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:40:01.194600 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:40:01.194603 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:40:01.194771 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:40:01.195395 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:40:01.195400 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:40:01.195607 (+ 207us) client_negotiation.cc:769] Sending connection context
0925 10:40:01.196439 (+ 832us) client_negotiation.cc:240] Negotiation successful
0925 10:40:01.196717 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":440,"threads_started":1}
I20260925 10:40:01.197654 23952 external_mini_cluster.cc:1428] Setting key 111c362687fa92022c5e78d65b6c9d14
WARNING: no policy specified for oryx/127.23.100.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.100.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/krb5kdc/oryx_127.23.100.3.keytab.
Entry for principal oryx/127.23.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/krb5kdc/oryx_127.23.100.3.keytab.
I20260925 10:40:01.239641 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.3:0
--local_ip_for_outbound_sockets=127.23.100.3
--webserver_interface=127.23.100.3
--webserver_port=0
--tserver_master_addrs=127.23.100.62:45071
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/krb5kdc/oryx_127.23.100.3.keytab
--principal=oryx/127.23.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:45727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:40:01.553972 31895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:40:01.554219 31895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:40:01.554423 31895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:40:01.561558 31895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:40:01.561672 31895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:40:01.561827 31895 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 10:40:01.561944 31895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.3
I20260925 10:40:01.571328 31895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:45727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/krb5kdc/oryx_127.23.100.3.keytab
--principal=oryx/127.23.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.100.62:45071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.3
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:40:01.572463 31895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:40:01.573814 31895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:40:01.586691 31901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:40:01.586655 31900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:40:01.586721 31903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:40:01.587355 31895 server_base.cc:1034] running on GCE node
I20260925 10:40:01.587970 31895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:40:01.588793 31895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:40:01.590030 31895 hybrid_clock.cc:648] HybridClock initialized: now 1790332801589979 us; error 51 us; skew 500 ppm
I20260925 10:40:01.590353 31895 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:40:01 dist-test-slave-dxdd krb5kdc[31500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332801, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:40:01.593304 31895 init.cc:377] Logged in from keytab as oryx/127.23.100.3@KRBTEST.COM (short username oryx)
I20260925 10:40:01.594666 31895 webserver.cc:460] Webserver started at http://127.23.100.3:41847/ using document root <none> and password file <none>
I20260925 10:40:01.595311 31895 fs_manager.cc:359] Metadata directory not provided
I20260925 10:40:01.595463 31895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:40:01.595784 31895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:40:01.598488 31895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-2/data/instance:
uuid: "6a90537cc4ed423b975af8b9cd87f711"
format_stamp: "Formatted at 2026-09-25 10:40:01 on dist-test-slave-dxdd"
server_key: "7a9567d5bcd88fd5c36c9a320f4e77e2"
server_key_iv: "efcbe18a55e4d6393f239e57aa21363d"
server_key_version: "encryptionkey@0"
I20260925 10:40:01.599241 31895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-2/wal/instance:
uuid: "6a90537cc4ed423b975af8b9cd87f711"
format_stamp: "Formatted at 2026-09-25 10:40:01 on dist-test-slave-dxdd"
server_key: "7a9567d5bcd88fd5c36c9a320f4e77e2"
server_key_iv: "efcbe18a55e4d6393f239e57aa21363d"
server_key_version: "encryptionkey@0"
I20260925 10:40:01.603957 31895 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 10:40:01.607115 31910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:40:01.608260 31895 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 10:40:01.608423 31895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-2/wal
uuid: "6a90537cc4ed423b975af8b9cd87f711"
format_stamp: "Formatted at 2026-09-25 10:40:01 on dist-test-slave-dxdd"
server_key: "7a9567d5bcd88fd5c36c9a320f4e77e2"
server_key_iv: "efcbe18a55e4d6393f239e57aa21363d"
server_key_version: "encryptionkey@0"
I20260925 10:40:01.608619 31895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:40:01.619797 31895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:40:01.624106 31895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:40:01.624492 31895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:40:01.627277 31895 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:40:01.627411 31895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:40:01.627591 31895 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:40:01.627686 31895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 10:40:01 dist-test-slave-dxdd krb5kdc[31500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332801, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.3@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM
I20260925 10:40:01.651722 31640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:01.627978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:50575 (local address 127.23.100.62:45071)
0925 10:40:01.628163 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:01.628169 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:01.629457 (+ 1288us) server_negotiation.cc:407] Connection header received
0925 10:40:01.631745 (+ 2288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:01.631754 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:01.631874 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:01.632008 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:01.633946 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:01.634896 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:01.638351 (+ 3455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:01.638961 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:01.639123 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:01.643914 (+ 4791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:40:01.643931 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:40:01.643937 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:40:01.643986 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:40:01.648116 (+ 4130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:40:01.649007 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:40:01.649011 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:40:01.649014 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:40:01.649135 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:40:01.649705 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:40:01.649712 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:40:01.649717 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:40:01.650009 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:40:01.650172 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:01.651136 (+ 964us) server_negotiation.cc:299] Negotiation successful
0925 10:40:01.651358 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 10:40:01.652936 31919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:01.628441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:45071 (local address 127.23.100.3:50575)
0925 10:40:01.629270 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:01.629317 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:01.631374 (+ 2057us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:01.632310 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:01.632320 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:01.633033 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:01.633770 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:01.633791 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:01.635140 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:01.635148 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:01.638147 (+ 2999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:01.638162 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:01.639458 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:01.639464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:01.639697 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:01.640386 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:01.640438 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:01.643588 (+ 3150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:40:01.648263 (+ 4675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:40:01.648269 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:40:01.648279 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:40:01.648821 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:40:01.649265 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:40:01.649268 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:40:01.649270 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:40:01.649519 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:40:01.650153 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:40:01.650160 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:40:01.650634 (+ 474us) client_negotiation.cc:769] Sending connection context
0925 10:40:01.651031 (+ 397us) client_negotiation.cc:240] Negotiation successful
0925 10:40:01.651538 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":367,"threads_started":1}
I20260925 10:40:01.670585 31895 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.3:37461
I20260925 10:40:01.670645 32029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.3:37461 every 8 connection(s)
I20260925 10:40:01.671777 31895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
I20260925 10:40:01.673846 32003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:01.659774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:45071 (local address 127.23.100.3:51197)
0925 10:40:01.660377 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:01.660392 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:01.660545 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:01.660872 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:01.660876 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:01.661587 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:01.661903 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:01.661915 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:01.663067 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:01.663073 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:01.664736 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:01.664754 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:01.665726 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:01.665737 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:01.665887 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:01.666679 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:01.666702 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:01.667897 (+ 1195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:40:01.671209 (+ 3312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:40:01.671215 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:40:01.671218 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:40:01.671578 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:40:01.671924 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:40:01.671931 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:40:01.671934 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:40:01.672058 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:40:01.672726 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:40:01.672733 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:40:01.672947 (+ 214us) client_negotiation.cc:769] Sending connection context
0925 10:40:01.673102 (+ 155us) client_negotiation.cc:240] Negotiation successful
0925 10:40:01.673388 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":347,"threads_started":1}
I20260925 10:40:01.673945 32030 heartbeater.cc:347] Connected to a master server at 127.23.100.62:45071
I20260925 10:40:01.674153 32030 heartbeater.cc:464] Registering TS with master...
I20260925 10:40:01.674731 31640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:01.659630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:51197 (local address 127.23.100.62:45071)
0925 10:40:01.659773 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:01.659776 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:01.660481 (+ 705us) server_negotiation.cc:407] Connection header received
0925 10:40:01.660685 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:01.660690 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:01.660753 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:01.660849 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:01.662044 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:01.662930 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:01.664941 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:01.665535 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:01.665668 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:01.668125 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:40:01.668133 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:40:01.668142 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:40:01.668171 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:40:01.671078 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:40:01.671700 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:40:01.671710 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:40:01.671712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:40:01.671819 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:40:01.672170 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:40:01.672175 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:40:01.672178 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:40:01.672494 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:40:01.672654 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:01.674132 (+ 1478us) server_negotiation.cc:299] Negotiation successful
0925 10:40:01.674377 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 10:40:01.675182 32030 heartbeater.cc:511] Master 127.23.100.62:45071 requested a full tablet report, sending...
I20260925 10:40:01.676508 31545 ts_manager.cc:194] Registered new tserver with Master: 6a90537cc4ed423b975af8b9cd87f711 (127.23.100.3:37461)
I20260925 10:40:01.677047 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 31895
I20260925 10:40:01.677416 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-2/wal/instance
I20260925 10:40:01.677560 31545 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.100.3@KRBTEST.COM'} at 127.23.100.3:50575
I20260925 10:40:01.678064 23952 external_mini_cluster.cc:1428] Setting key 50bf4dff96f2a5ffe946b01825645dc8
I20260925 10:40:01.680816 23952 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 10:40:01 dist-test-slave-dxdd krb5kdc[31500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332801, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260925 10:40:01.710498 31752 heartbeater.cc:503] Master 127.23.100.62:45071 was elected leader, sending a full tablet report...
Sep 25 10:40:01 dist-test-slave-dxdd krb5kdc[31500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:40:01 dist-test-slave-dxdd krb5kdc[31500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:40:02.199378 31891 heartbeater.cc:503] Master 127.23.100.62:45071 was elected leader, sending a full tablet report...
I20260925 10:40:02.680330 32030 heartbeater.cc:503] Master 127.23.100.62:45071 was elected leader, sending a full tablet report...
W20260925 10:40:02.764441 32038 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.042s user 0.004s sys 0.009s
W20260925 10:40:02.765714 31640 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:40:01.717443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48574 (local address 127.23.100.62:45071)
0925 10:40:01.717659 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:01.717666 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:01.717916 (+ 250us) server_negotiation.cc:407] Connection header received
0925 10:40:01.718048 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:01.718052 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:01.718109 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:01.718188 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:01.719474 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:01.720253 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:01.721393 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:01.721784 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:01.721871 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:02.765457 (+1043586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48574: BlockingRecv error: recv got EOF from 127.0.0.1:48574 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 25 10:40:02 dist-test-slave-dxdd krb5kdc[31500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332801, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM
I20260925 10:40:02.786962 31640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:02.768998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48582 (local address 127.23.100.62:45071)
0925 10:40:02.769150 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:02.769155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:02.769534 (+ 379us) server_negotiation.cc:407] Connection header received
0925 10:40:02.769656 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:02.769659 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:02.769749 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:02.769830 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:02.771234 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:02.772017 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:02.773309 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:02.773766 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:02.773856 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:02.781727 (+ 7871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:40:02.781731 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:40:02.781735 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:40:02.781764 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:40:02.784818 (+ 3054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:40:02.785404 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:40:02.785408 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:40:02.785410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:40:02.785521 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:40:02.785822 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:40:02.785825 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:40:02.785827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:40:02.786079 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:40:02.786298 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:02.786573 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 10:40:02.786731 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 10:40:02.790697 31545 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48582:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:40:02.797329 31545 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:40:02.820189 32049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:02.808505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.3:37461 (local address 127.0.0.1:38188)
0925 10:40:02.809498 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:02.809552 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:02.809729 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:02.811418 (+ 1689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:02.811429 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:02.811465 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:40:02.812102 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:02.812123 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:02.815039 (+ 2916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:02.815047 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:02.817536 (+ 2489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:02.817557 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:02.819099 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:02.819105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:02.819269 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:02.819325 (+ 56us) client_negotiation.cc:769] Sending connection context
0925 10:40:02.819515 (+ 190us) client_negotiation.cc:240] Negotiation successful
0925 10:40:02.819684 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"mutex_wait_us":22,"thread_start_us":454,"threads_started":1}
I20260925 10:40:02.820827 32050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:02.808735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38188 (local address 127.23.100.3:37461)
0925 10:40:02.810678 (+ 1943us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:02.810686 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:02.810713 (+ 27us) server_negotiation.cc:407] Connection header received
0925 10:40:02.810824 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:02.810835 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:02.811124 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:02.812416 (+ 1292us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:40:02.812698 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:02.814858 (+ 2160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:02.817756 (+ 2898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:02.818907 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:02.819055 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:02.819256 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:02.820172 (+ 916us) server_negotiation.cc:299] Negotiation successful
0925 10:40:02.820510 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1792,"thread_start_us":1831,"threads_started":1}
I20260925 10:40:02.821218 32052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:02.809478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51634 (local address 127.23.100.1:37725)
0925 10:40:02.810227 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:02.810239 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:02.810984 (+ 745us) server_negotiation.cc:407] Connection header received
0925 10:40:02.811097 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:02.811108 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:02.811391 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:02.811640 (+ 249us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:40:02.812660 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:02.814631 (+ 1971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:02.818116 (+ 3485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:02.819824 (+ 1708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:02.820037 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:02.820243 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:02.820747 (+ 504us) server_negotiation.cc:299] Negotiation successful
0925 10:40:02.820931 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":449,"threads_started":1}
I20260925 10:40:02.822114 32051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:02.808881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:37725 (local address 127.0.0.1:51634)
0925 10:40:02.810722 (+ 1841us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:02.810738 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:02.810857 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:02.811923 (+ 1066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:02.811930 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:02.811985 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:40:02.812445 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:02.812458 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:02.814831 (+ 2373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:02.814839 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:02.817972 (+ 3133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:02.817984 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:02.820382 (+ 2398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:02.820388 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:02.820562 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:02.820593 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 10:40:02.820692 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 10:40:02.820809 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1714,"mutex_wait_us":1,"thread_start_us":1972,"threads_started":1}
I20260925 10:40:02.824558 31960 tablet_service.cc:1467] Processing CreateTablet for tablet 2906618b3b7948a69e5d3090961d8ab8 (DEFAULT_TABLE table=test-table [id=be525b209a4446d6900bb9a690512fe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:40:02.825132 32053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:02.810474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.2:35365 (local address 127.0.0.1:33060)
0925 10:40:02.816083 (+ 5609us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:02.816098 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:02.816200 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:02.817572 (+ 1372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:02.817576 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:02.817604 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:40:02.818026 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:02.818036 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:02.819825 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:02.819851 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:02.822016 (+ 2165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:02.822030 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:02.824366 (+ 2336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:02.824372 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:02.824551 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:02.824584 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 10:40:02.824670 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 10:40:02.824790 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5494,"mutex_wait_us":1,"thread_start_us":5364,"threads_started":1}
I20260925 10:40:02.825614 32054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:02.810716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33060 (local address 127.23.100.2:35365)
0925 10:40:02.817073 (+ 6357us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:02.817080 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:02.817105 (+ 25us) server_negotiation.cc:407] Connection header received
0925 10:40:02.817195 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:02.817202 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:02.817389 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:02.817571 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:40:02.818214 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:02.819598 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:02.822252 (+ 2654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:02.823905 (+ 1653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:02.824055 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:02.824233 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:02.825177 (+ 944us) server_negotiation.cc:299] Negotiation successful
0925 10:40:02.825321 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6198,"thread_start_us":6056,"threads_started":1}
I20260925 10:40:02.825829 31960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2906618b3b7948a69e5d3090961d8ab8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:40:02.825562 31682 tablet_service.cc:1467] Processing CreateTablet for tablet 2906618b3b7948a69e5d3090961d8ab8 (DEFAULT_TABLE table=test-table [id=be525b209a4446d6900bb9a690512fe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:40:02.826951 31682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2906618b3b7948a69e5d3090961d8ab8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:40:02.828878 31821 tablet_service.cc:1467] Processing CreateTablet for tablet 2906618b3b7948a69e5d3090961d8ab8 (DEFAULT_TABLE table=test-table [id=be525b209a4446d6900bb9a690512fe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:40:02.830137 31821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2906618b3b7948a69e5d3090961d8ab8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:40:02.836862 32055 tablet_bootstrap.cc:492] T 2906618b3b7948a69e5d3090961d8ab8 P 6a90537cc4ed423b975af8b9cd87f711: Bootstrap starting.
I20260925 10:40:02.839747 32055 tablet_bootstrap.cc:654] T 2906618b3b7948a69e5d3090961d8ab8 P 6a90537cc4ed423b975af8b9cd87f711: Neither blocks nor log segments found. Creating new log.
I20260925 10:40:02.841022 32056 tablet_bootstrap.cc:492] T 2906618b3b7948a69e5d3090961d8ab8 P dcafa1392bf049e4b508fb1ef2a74889: Bootstrap starting.
I20260925 10:40:02.841759 32055 log.cc:824] T 2906618b3b7948a69e5d3090961d8ab8 P 6a90537cc4ed423b975af8b9cd87f711: Log is configured to *not* fsync() on all Append() calls
I20260925 10:40:02.844262 32056 tablet_bootstrap.cc:654] T 2906618b3b7948a69e5d3090961d8ab8 P dcafa1392bf049e4b508fb1ef2a74889: Neither blocks nor log segments found. Creating new log.
I20260925 10:40:02.844795 32055 tablet_bootstrap.cc:492] T 2906618b3b7948a69e5d3090961d8ab8 P 6a90537cc4ed423b975af8b9cd87f711: No bootstrap required, opened a new log
I20260925 10:40:02.845045 32058 tablet_bootstrap.cc:492] T 2906618b3b7948a69e5d3090961d8ab8 P d78e073456484d9082e78231eee41e48: Bootstrap starting.
I20260925 10:40:02.845055 32055 ts_tablet_manager.cc:1397] T 2906618b3b7948a69e5d3090961d8ab8 P 6a90537cc4ed423b975af8b9cd87f711: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.004s
I20260925 10:40:02.845424 32056 log.cc:824] T 2906618b3b7948a69e5d3090961d8ab8 P dcafa1392bf049e4b508fb1ef2a74889: Log is configured to *not* fsync() on all Append() calls
I20260925 10:40:02.847848 32058 tablet_bootstrap.cc:654] T 2906618b3b7948a69e5d3090961d8ab8 P d78e073456484d9082e78231eee41e48: Neither blocks nor log segments found. Creating new log.
I20260925 10:40:02.848237 32056 tablet_bootstrap.cc:492] T 2906618b3b7948a69e5d3090961d8ab8 P dcafa1392bf049e4b508fb1ef2a74889: No bootstrap required, opened a new log
I20260925 10:40:02.848459 32056 ts_tablet_manager.cc:1397] T 2906618b3b7948a69e5d3090961d8ab8 P dcafa1392bf049e4b508fb1ef2a74889: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260925 10:40:02.848834 32058 log.cc:824] T 2906618b3b7948a69e5d3090961d8ab8 P d78e073456484d9082e78231eee41e48: Log is configured to *not* fsync() on all Append() calls
I20260925 10:40:02.851959 32058 tablet_bootstrap.cc:492] T 2906618b3b7948a69e5d3090961d8ab8 P d78e073456484d9082e78231eee41e48: No bootstrap required, opened a new log
I20260925 10:40:02.852214 32058 ts_tablet_manager.cc:1397] T 2906618b3b7948a69e5d3090961d8ab8 P d78e073456484d9082e78231eee41e48: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 10:40:02.855629 32055 raft_consensus.cc:348] T 2906618b3b7948a69e5d3090961d8ab8 P 6a90537cc4ed423b975af8b9cd87f711 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } } peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } }
I20260925 10:40:02.856007 32055 raft_consensus.cc:374] T 2906618b3b7948a69e5d3090961d8ab8 P 6a90537cc4ed423b975af8b9cd87f711 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:40:02.856096 32055 raft_consensus.cc:729] T 2906618b3b7948a69e5d3090961d8ab8 P 6a90537cc4ed423b975af8b9cd87f711 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a90537cc4ed423b975af8b9cd87f711, State: Initialized, Role: FOLLOWER
I20260925 10:40:02.856662 32055 consensus_queue.cc:260] T 2906618b3b7948a69e5d3090961d8ab8 P 6a90537cc4ed423b975af8b9cd87f711 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } } peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } }
I20260925 10:40:02.858212 32055 ts_tablet_manager.cc:1428] T 2906618b3b7948a69e5d3090961d8ab8 P 6a90537cc4ed423b975af8b9cd87f711: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s
I20260925 10:40:02.858448 32056 raft_consensus.cc:348] T 2906618b3b7948a69e5d3090961d8ab8 P dcafa1392bf049e4b508fb1ef2a74889 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } } peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } }
I20260925 10:40:02.858803 32056 raft_consensus.cc:374] T 2906618b3b7948a69e5d3090961d8ab8 P dcafa1392bf049e4b508fb1ef2a74889 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:40:02.858904 32056 raft_consensus.cc:729] T 2906618b3b7948a69e5d3090961d8ab8 P dcafa1392bf049e4b508fb1ef2a74889 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcafa1392bf049e4b508fb1ef2a74889, State: Initialized, Role: FOLLOWER
I20260925 10:40:02.859447 32056 consensus_queue.cc:260] T 2906618b3b7948a69e5d3090961d8ab8 P dcafa1392bf049e4b508fb1ef2a74889 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } } peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } }
I20260925 10:40:02.860718 32058 raft_consensus.cc:348] T 2906618b3b7948a69e5d3090961d8ab8 P d78e073456484d9082e78231eee41e48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } } peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } }
I20260925 10:40:02.861148 32058 raft_consensus.cc:374] T 2906618b3b7948a69e5d3090961d8ab8 P d78e073456484d9082e78231eee41e48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:40:02.861271 32058 raft_consensus.cc:729] T 2906618b3b7948a69e5d3090961d8ab8 P d78e073456484d9082e78231eee41e48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d78e073456484d9082e78231eee41e48, State: Initialized, Role: FOLLOWER
I20260925 10:40:02.861800 32056 ts_tablet_manager.cc:1428] T 2906618b3b7948a69e5d3090961d8ab8 P dcafa1392bf049e4b508fb1ef2a74889: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260925 10:40:02.862037 32058 consensus_queue.cc:260] T 2906618b3b7948a69e5d3090961d8ab8 P d78e073456484d9082e78231eee41e48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } } peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } }
I20260925 10:40:02.863596 32058 ts_tablet_manager.cc:1428] T 2906618b3b7948a69e5d3090961d8ab8 P d78e073456484d9082e78231eee41e48: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
W20260925 10:40:02.924113 32031 tablet.cc:2369] T 2906618b3b7948a69e5d3090961d8ab8 P 6a90537cc4ed423b975af8b9cd87f711: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:40:02.944080 31892 tablet.cc:2369] T 2906618b3b7948a69e5d3090961d8ab8 P d78e073456484d9082e78231eee41e48: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:40:02.954445 31753 tablet.cc:2369] T 2906618b3b7948a69e5d3090961d8ab8 P dcafa1392bf049e4b508fb1ef2a74889: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:40:03.101784 32063 raft_consensus.cc:482] T 2906618b3b7948a69e5d3090961d8ab8 P d78e073456484d9082e78231eee41e48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:40:03.102110 32063 raft_consensus.cc:504] T 2906618b3b7948a69e5d3090961d8ab8 P d78e073456484d9082e78231eee41e48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } } peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } }
I20260925 10:40:03.103817 32063 leader_election.cc:288] T 2906618b3b7948a69e5d3090961d8ab8 P d78e073456484d9082e78231eee41e48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a90537cc4ed423b975af8b9cd87f711 (127.23.100.3:37461), dcafa1392bf049e4b508fb1ef2a74889 (127.23.100.1:37725)
I20260925 10:40:03.111222 32064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:03.104170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.3:37461 (local address 127.23.100.2:51493)
0925 10:40:03.105151 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:03.105163 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:03.105260 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:03.105596 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:03.105600 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:03.105622 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:40:03.105973 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:03.105992 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:03.107007 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:03.107014 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:03.109450 (+ 2436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:03.109459 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:03.110605 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:03.110609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:03.110707 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:03.110730 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 10:40:03.110807 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 10:40:03.110889 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":880,"mutex_wait_us":188,"thread_start_us":498,"threads_started":1}
I20260925 10:40:03.111225 32050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:03.104226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:51493 (local address 127.23.100.3:37461)
0925 10:40:03.104465 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:03.104474 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:03.105250 (+ 776us) server_negotiation.cc:407] Connection header received
0925 10:40:03.105428 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:03.105431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:03.105486 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:03.105583 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:40:03.106145 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:03.106867 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:03.109599 (+ 2732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:03.110429 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:03.110587 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:03.110659 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:03.110852 (+ 193us) server_negotiation.cc:299] Negotiation successful
0925 10:40:03.110984 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 10:40:03.112260 31980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2906618b3b7948a69e5d3090961d8ab8" candidate_uuid: "d78e073456484d9082e78231eee41e48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a90537cc4ed423b975af8b9cd87f711" is_pre_election: true
I20260925 10:40:03.112540 32052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:03.104426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:42991 (local address 127.23.100.1:37725)
0925 10:40:03.104660 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:03.104668 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:03.105138 (+ 470us) server_negotiation.cc:407] Connection header received
0925 10:40:03.105400 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:03.105404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:03.105487 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:03.105581 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:40:03.106228 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:03.106970 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:03.110429 (+ 3459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:03.111695 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:03.111855 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:03.111904 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:03.112230 (+ 326us) server_negotiation.cc:299] Negotiation successful
0925 10:40:03.112359 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 10:40:03.112712 32065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:03.104055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:37725 (local address 127.23.100.2:42991)
0925 10:40:03.105029 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:03.105043 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:03.105214 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:03.105610 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:03.105615 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:03.105638 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:40:03.106056 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:03.106088 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:03.107111 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:03.107114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:03.109446 (+ 2332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:03.109459 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:03.111903 (+ 2444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:03.111908 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:03.112065 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:03.112097 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 10:40:03.112196 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 10:40:03.112326 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":831,"mutex_wait_us":54,"thread_start_us":853,"threads_started":1}
I20260925 10:40:03.112808 31980 raft_consensus.cc:2454] T 2906618b3b7948a69e5d3090961d8ab8 P 6a90537cc4ed423b975af8b9cd87f711 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d78e073456484d9082e78231eee41e48 in term 0.
I20260925 10:40:03.113493 31774 leader_election.cc:302] T 2906618b3b7948a69e5d3090961d8ab8 P d78e073456484d9082e78231eee41e48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a90537cc4ed423b975af8b9cd87f711, d78e073456484d9082e78231eee41e48; no voters:
I20260925 10:40:03.113912 32063 raft_consensus.cc:2781] T 2906618b3b7948a69e5d3090961d8ab8 P d78e073456484d9082e78231eee41e48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:40:03.113873 31702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2906618b3b7948a69e5d3090961d8ab8" candidate_uuid: "d78e073456484d9082e78231eee41e48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcafa1392bf049e4b508fb1ef2a74889" is_pre_election: true
I20260925 10:40:03.114059 32063 raft_consensus.cc:482] T 2906618b3b7948a69e5d3090961d8ab8 P d78e073456484d9082e78231eee41e48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:40:03.114182 32063 raft_consensus.cc:3037] T 2906618b3b7948a69e5d3090961d8ab8 P d78e073456484d9082e78231eee41e48 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:40:03.114408 31702 raft_consensus.cc:2454] T 2906618b3b7948a69e5d3090961d8ab8 P dcafa1392bf049e4b508fb1ef2a74889 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d78e073456484d9082e78231eee41e48 in term 0.
I20260925 10:40:03.116617 32063 raft_consensus.cc:504] T 2906618b3b7948a69e5d3090961d8ab8 P d78e073456484d9082e78231eee41e48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } } peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } }
I20260925 10:40:03.117230 32063 leader_election.cc:288] T 2906618b3b7948a69e5d3090961d8ab8 P d78e073456484d9082e78231eee41e48 [CANDIDATE]: Term 1 election: Requested vote from peers 6a90537cc4ed423b975af8b9cd87f711 (127.23.100.3:37461), dcafa1392bf049e4b508fb1ef2a74889 (127.23.100.1:37725)
I20260925 10:40:03.117558 31980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2906618b3b7948a69e5d3090961d8ab8" candidate_uuid: "d78e073456484d9082e78231eee41e48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a90537cc4ed423b975af8b9cd87f711"
I20260925 10:40:03.117642 31702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2906618b3b7948a69e5d3090961d8ab8" candidate_uuid: "d78e073456484d9082e78231eee41e48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcafa1392bf049e4b508fb1ef2a74889"
I20260925 10:40:03.117731 31980 raft_consensus.cc:3037] T 2906618b3b7948a69e5d3090961d8ab8 P 6a90537cc4ed423b975af8b9cd87f711 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:40:03.117869 31702 raft_consensus.cc:3037] T 2906618b3b7948a69e5d3090961d8ab8 P dcafa1392bf049e4b508fb1ef2a74889 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:40:03.119915 31980 raft_consensus.cc:2454] T 2906618b3b7948a69e5d3090961d8ab8 P 6a90537cc4ed423b975af8b9cd87f711 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d78e073456484d9082e78231eee41e48 in term 1.
I20260925 10:40:03.120170 31702 raft_consensus.cc:2454] T 2906618b3b7948a69e5d3090961d8ab8 P dcafa1392bf049e4b508fb1ef2a74889 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d78e073456484d9082e78231eee41e48 in term 1.
I20260925 10:40:03.120321 31774 leader_election.cc:302] T 2906618b3b7948a69e5d3090961d8ab8 P d78e073456484d9082e78231eee41e48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a90537cc4ed423b975af8b9cd87f711, d78e073456484d9082e78231eee41e48; no voters:
I20260925 10:40:03.120597 32063 raft_consensus.cc:2781] T 2906618b3b7948a69e5d3090961d8ab8 P d78e073456484d9082e78231eee41e48 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:40:03.120965 32063 raft_consensus.cc:686] T 2906618b3b7948a69e5d3090961d8ab8 P d78e073456484d9082e78231eee41e48 [term 1 LEADER]: Becoming Leader. State: Replica: d78e073456484d9082e78231eee41e48, State: Running, Role: LEADER
I20260925 10:40:03.121407 32063 consensus_queue.cc:237] T 2906618b3b7948a69e5d3090961d8ab8 P d78e073456484d9082e78231eee41e48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } } peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } }
I20260925 10:40:03.125918 31544 catalog_manager.cc:5495] T 2906618b3b7948a69e5d3090961d8ab8 P d78e073456484d9082e78231eee41e48 reported cstate change: term changed from 0 to 1, leader changed from <none> to d78e073456484d9082e78231eee41e48 (127.23.100.2). New cstate: current_term: 1 leader_uuid: "d78e073456484d9082e78231eee41e48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } health_report { overall_health: HEALTHY } } }
I20260925 10:40:03.184511 32054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:03.179580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33072 (local address 127.23.100.2:35365)
0925 10:40:03.179814 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:03.179818 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:03.179835 (+ 17us) server_negotiation.cc:407] Connection header received
0925 10:40:03.180077 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:03.180081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:03.180161 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:03.180279 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:40:03.180568 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:03.181242 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:03.182457 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:03.182840 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:03.182925 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:03.183315 (+ 390us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:40:03.183947 (+ 632us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:40:03.184047 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:03.184148 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 10:40:03.184212 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 10:40:03.192354 31980 raft_consensus.cc:1261] T 2906618b3b7948a69e5d3090961d8ab8 P 6a90537cc4ed423b975af8b9cd87f711 [term 1 FOLLOWER]: Refusing update from remote peer d78e073456484d9082e78231eee41e48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:40:03.192518 31702 raft_consensus.cc:1261] T 2906618b3b7948a69e5d3090961d8ab8 P dcafa1392bf049e4b508fb1ef2a74889 [term 1 FOLLOWER]: Refusing update from remote peer d78e073456484d9082e78231eee41e48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:40:03.193413 32063 consensus_queue.cc:1035] T 2906618b3b7948a69e5d3090961d8ab8 P d78e073456484d9082e78231eee41e48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:40:03.193991 32066 consensus_queue.cc:1035] T 2906618b3b7948a69e5d3090961d8ab8 P d78e073456484d9082e78231eee41e48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:40:03.208566 32069 mvcc.cc:205] Tried to move back new op lower bound from 7333203161865621504 to 7333203161588527104. Current Snapshot: MvccSnapshot[applied={T|T < 7333203161865621504}]
I20260925 10:40:03.220789 31545 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:40:03.221045 31545 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:40:03.224565 31545 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2620a3124bff4655ac88451061702e7d: Sending DeleteTablet for 3 replicas of tablet 2906618b3b7948a69e5d3090961d8ab8
I20260925 10:40:03.225561 31960 tablet_service.cc:1514] Processing DeleteTablet for tablet 2906618b3b7948a69e5d3090961d8ab8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:40:03 UTC) from {username='oryx', principal='oryx/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:38188
I20260925 10:40:03.225680 31682 tablet_service.cc:1514] Processing DeleteTablet for tablet 2906618b3b7948a69e5d3090961d8ab8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:40:03 UTC) from {username='oryx', principal='oryx/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:51634
I20260925 10:40:03.225999 31821 tablet_service.cc:1514] Processing DeleteTablet for tablet 2906618b3b7948a69e5d3090961d8ab8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:40:03 UTC) from {username='oryx', principal='oryx/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:33060
I20260925 10:40:03.228543 31545 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48582:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:40:03.229370 31545 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:40:03.238844 32080 tablet_replica.cc:330] T 2906618b3b7948a69e5d3090961d8ab8 P dcafa1392bf049e4b508fb1ef2a74889: stopping tablet replica
I20260925 10:40:03.239403 32080 raft_consensus.cc:2229] T 2906618b3b7948a69e5d3090961d8ab8 P dcafa1392bf049e4b508fb1ef2a74889 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:40:03.239988 32080 raft_consensus.cc:2258] T 2906618b3b7948a69e5d3090961d8ab8 P dcafa1392bf049e4b508fb1ef2a74889 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:40:03.242280 32080 ts_tablet_manager.cc:1905] T 2906618b3b7948a69e5d3090961d8ab8 P dcafa1392bf049e4b508fb1ef2a74889: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:40:03.243314 31959 tablet_service.cc:1467] Processing CreateTablet for tablet 3232251de2a44dfeabae71ef64de3d0c (DEFAULT_TABLE table=test-table [id=98b803824b3142289325b194728aeeae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:40:03.244050 31959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3232251de2a44dfeabae71ef64de3d0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:40:03.250296 32079 tablet_replica.cc:330] T 2906618b3b7948a69e5d3090961d8ab8 P 6a90537cc4ed423b975af8b9cd87f711: stopping tablet replica
I20260925 10:40:03.250854 32079 raft_consensus.cc:2229] T 2906618b3b7948a69e5d3090961d8ab8 P 6a90537cc4ed423b975af8b9cd87f711 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:40:03.250921 31820 tablet_service.cc:1467] Processing CreateTablet for tablet 3232251de2a44dfeabae71ef64de3d0c (DEFAULT_TABLE table=test-table [id=98b803824b3142289325b194728aeeae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:40:03.251430 32079 raft_consensus.cc:2258] T 2906618b3b7948a69e5d3090961d8ab8 P 6a90537cc4ed423b975af8b9cd87f711 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:40:03.251493 31820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3232251de2a44dfeabae71ef64de3d0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:40:03.252287 32081 tablet_replica.cc:330] T 2906618b3b7948a69e5d3090961d8ab8 P d78e073456484d9082e78231eee41e48: stopping tablet replica
I20260925 10:40:03.252416 31682 tablet_service.cc:1467] Processing CreateTablet for tablet 3232251de2a44dfeabae71ef64de3d0c (DEFAULT_TABLE table=test-table [id=98b803824b3142289325b194728aeeae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:40:03.252815 31682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3232251de2a44dfeabae71ef64de3d0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:40:03.254868 32055 tablet_bootstrap.cc:492] T 3232251de2a44dfeabae71ef64de3d0c P 6a90537cc4ed423b975af8b9cd87f711: Bootstrap starting.
I20260925 10:40:03.256151 32079 ts_tablet_manager.cc:1905] T 2906618b3b7948a69e5d3090961d8ab8 P 6a90537cc4ed423b975af8b9cd87f711: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:40:03.256484 32055 tablet_bootstrap.cc:654] T 3232251de2a44dfeabae71ef64de3d0c P 6a90537cc4ed423b975af8b9cd87f711: Neither blocks nor log segments found. Creating new log.
I20260925 10:40:03.258316 32058 tablet_bootstrap.cc:492] T 3232251de2a44dfeabae71ef64de3d0c P d78e073456484d9082e78231eee41e48: Bootstrap starting.
I20260925 10:40:03.258898 32081 raft_consensus.cc:2229] T 2906618b3b7948a69e5d3090961d8ab8 P d78e073456484d9082e78231eee41e48 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:40:03.259603 32081 raft_consensus.cc:2258] T 2906618b3b7948a69e5d3090961d8ab8 P d78e073456484d9082e78231eee41e48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:40:03.260252 32058 tablet_bootstrap.cc:654] T 3232251de2a44dfeabae71ef64de3d0c P d78e073456484d9082e78231eee41e48: Neither blocks nor log segments found. Creating new log.
I20260925 10:40:03.261554 32081 ts_tablet_manager.cc:1905] T 2906618b3b7948a69e5d3090961d8ab8 P d78e073456484d9082e78231eee41e48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:40:03.263201 32079 ts_tablet_manager.cc:1918] T 2906618b3b7948a69e5d3090961d8ab8 P 6a90537cc4ed423b975af8b9cd87f711: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:40:03.263370 32079 log.cc:1196] T 2906618b3b7948a69e5d3090961d8ab8 P 6a90537cc4ed423b975af8b9cd87f711: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-2/wal/wals/2906618b3b7948a69e5d3090961d8ab8
I20260925 10:40:03.263931 32079 ts_tablet_manager.cc:1939] T 2906618b3b7948a69e5d3090961d8ab8 P 6a90537cc4ed423b975af8b9cd87f711: Deleting consensus metadata
I20260925 10:40:03.265725 31531 catalog_manager.cc:4841] TS 6a90537cc4ed423b975af8b9cd87f711 (127.23.100.3:37461): tablet 2906618b3b7948a69e5d3090961d8ab8 (table test-table [id=be525b209a4446d6900bb9a690512fe1]) successfully deleted
I20260925 10:40:03.266646 32080 ts_tablet_manager.cc:1918] T 2906618b3b7948a69e5d3090961d8ab8 P dcafa1392bf049e4b508fb1ef2a74889: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:40:03.266803 32080 log.cc:1196] T 2906618b3b7948a69e5d3090961d8ab8 P dcafa1392bf049e4b508fb1ef2a74889: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-0/wal/wals/2906618b3b7948a69e5d3090961d8ab8
I20260925 10:40:03.267398 32080 ts_tablet_manager.cc:1939] T 2906618b3b7948a69e5d3090961d8ab8 P dcafa1392bf049e4b508fb1ef2a74889: Deleting consensus metadata
I20260925 10:40:03.269078 31529 catalog_manager.cc:4841] TS dcafa1392bf049e4b508fb1ef2a74889 (127.23.100.1:37725): tablet 2906618b3b7948a69e5d3090961d8ab8 (table test-table [id=be525b209a4446d6900bb9a690512fe1]) successfully deleted
I20260925 10:40:03.270886 32056 tablet_bootstrap.cc:492] T 3232251de2a44dfeabae71ef64de3d0c P dcafa1392bf049e4b508fb1ef2a74889: Bootstrap starting.
I20260925 10:40:03.272508 32058 tablet_bootstrap.cc:492] T 3232251de2a44dfeabae71ef64de3d0c P d78e073456484d9082e78231eee41e48: No bootstrap required, opened a new log
I20260925 10:40:03.272666 32056 tablet_bootstrap.cc:654] T 3232251de2a44dfeabae71ef64de3d0c P dcafa1392bf049e4b508fb1ef2a74889: Neither blocks nor log segments found. Creating new log.
I20260925 10:40:03.273466 32081 ts_tablet_manager.cc:1918] T 2906618b3b7948a69e5d3090961d8ab8 P d78e073456484d9082e78231eee41e48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:40:03.273633 32081 log.cc:1196] T 2906618b3b7948a69e5d3090961d8ab8 P d78e073456484d9082e78231eee41e48: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-1/wal/wals/2906618b3b7948a69e5d3090961d8ab8
I20260925 10:40:03.273835 32058 ts_tablet_manager.cc:1397] T 3232251de2a44dfeabae71ef64de3d0c P d78e073456484d9082e78231eee41e48: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.000s
I20260925 10:40:03.274259 32081 ts_tablet_manager.cc:1939] T 2906618b3b7948a69e5d3090961d8ab8 P d78e073456484d9082e78231eee41e48: Deleting consensus metadata
I20260925 10:40:03.274842 32058 raft_consensus.cc:348] T 3232251de2a44dfeabae71ef64de3d0c P d78e073456484d9082e78231eee41e48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } } peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } }
I20260925 10:40:03.275075 32058 raft_consensus.cc:374] T 3232251de2a44dfeabae71ef64de3d0c P d78e073456484d9082e78231eee41e48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:40:03.275200 32058 raft_consensus.cc:729] T 3232251de2a44dfeabae71ef64de3d0c P d78e073456484d9082e78231eee41e48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d78e073456484d9082e78231eee41e48, State: Initialized, Role: FOLLOWER
I20260925 10:40:03.275516 32058 consensus_queue.cc:260] T 3232251de2a44dfeabae71ef64de3d0c P d78e073456484d9082e78231eee41e48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } } peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } }
I20260925 10:40:03.275861 31531 catalog_manager.cc:4841] TS d78e073456484d9082e78231eee41e48 (127.23.100.2:35365): tablet 2906618b3b7948a69e5d3090961d8ab8 (table test-table [id=be525b209a4446d6900bb9a690512fe1]) successfully deleted
I20260925 10:40:03.276727 32055 tablet_bootstrap.cc:492] T 3232251de2a44dfeabae71ef64de3d0c P 6a90537cc4ed423b975af8b9cd87f711: No bootstrap required, opened a new log
I20260925 10:40:03.276889 32055 ts_tablet_manager.cc:1397] T 3232251de2a44dfeabae71ef64de3d0c P 6a90537cc4ed423b975af8b9cd87f711: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.004s
I20260925 10:40:03.277699 32055 raft_consensus.cc:348] T 3232251de2a44dfeabae71ef64de3d0c P 6a90537cc4ed423b975af8b9cd87f711 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } } peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } }
I20260925 10:40:03.277913 32055 raft_consensus.cc:374] T 3232251de2a44dfeabae71ef64de3d0c P 6a90537cc4ed423b975af8b9cd87f711 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:40:03.278035 32055 raft_consensus.cc:729] T 3232251de2a44dfeabae71ef64de3d0c P 6a90537cc4ed423b975af8b9cd87f711 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a90537cc4ed423b975af8b9cd87f711, State: Initialized, Role: FOLLOWER
I20260925 10:40:03.278353 32055 consensus_queue.cc:260] T 3232251de2a44dfeabae71ef64de3d0c P 6a90537cc4ed423b975af8b9cd87f711 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } } peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } }
I20260925 10:40:03.279034 32055 ts_tablet_manager.cc:1428] T 3232251de2a44dfeabae71ef64de3d0c P 6a90537cc4ed423b975af8b9cd87f711: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:40:03.279510 32058 ts_tablet_manager.cc:1428] T 3232251de2a44dfeabae71ef64de3d0c P d78e073456484d9082e78231eee41e48: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260925 10:40:03.283874 32056 tablet_bootstrap.cc:492] T 3232251de2a44dfeabae71ef64de3d0c P dcafa1392bf049e4b508fb1ef2a74889: No bootstrap required, opened a new log
I20260925 10:40:03.284022 32056 ts_tablet_manager.cc:1397] T 3232251de2a44dfeabae71ef64de3d0c P dcafa1392bf049e4b508fb1ef2a74889: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.008s
I20260925 10:40:03.284677 32056 raft_consensus.cc:348] T 3232251de2a44dfeabae71ef64de3d0c P dcafa1392bf049e4b508fb1ef2a74889 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } } peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } }
I20260925 10:40:03.284937 32056 raft_consensus.cc:374] T 3232251de2a44dfeabae71ef64de3d0c P dcafa1392bf049e4b508fb1ef2a74889 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:40:03.285044 32056 raft_consensus.cc:729] T 3232251de2a44dfeabae71ef64de3d0c P dcafa1392bf049e4b508fb1ef2a74889 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcafa1392bf049e4b508fb1ef2a74889, State: Initialized, Role: FOLLOWER
I20260925 10:40:03.285322 32056 consensus_queue.cc:260] T 3232251de2a44dfeabae71ef64de3d0c P dcafa1392bf049e4b508fb1ef2a74889 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } } peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } }
I20260925 10:40:03.286011 32056 ts_tablet_manager.cc:1428] T 3232251de2a44dfeabae71ef64de3d0c P dcafa1392bf049e4b508fb1ef2a74889: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260925 10:40:03.391985 32062 raft_consensus.cc:482] T 3232251de2a44dfeabae71ef64de3d0c P dcafa1392bf049e4b508fb1ef2a74889 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:40:03.392251 32062 raft_consensus.cc:504] T 3232251de2a44dfeabae71ef64de3d0c P dcafa1392bf049e4b508fb1ef2a74889 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } } peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } }
I20260925 10:40:03.393827 32062 leader_election.cc:288] T 3232251de2a44dfeabae71ef64de3d0c P dcafa1392bf049e4b508fb1ef2a74889 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d78e073456484d9082e78231eee41e48 (127.23.100.2:35365), 6a90537cc4ed423b975af8b9cd87f711 (127.23.100.3:37461)
I20260925 10:40:03.401665 32054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:03.394104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:42433 (local address 127.23.100.2:35365)
0925 10:40:03.394341 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:03.394350 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:03.394886 (+ 536us) server_negotiation.cc:407] Connection header received
0925 10:40:03.395230 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:03.395238 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:03.395360 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:03.395457 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:40:03.396284 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:03.397387 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:03.399399 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:03.400619 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:03.400785 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:03.400855 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:03.401192 (+ 337us) server_negotiation.cc:299] Negotiation successful
0925 10:40:03.401324 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 10:40:03.401696 32085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:03.394134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.2:35365 (local address 127.23.100.1:42433)
0925 10:40:03.394795 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:03.394815 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:03.395028 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:03.395697 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:03.395700 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:03.395720 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:40:03.396115 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:03.396127 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:03.397532 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:03.397537 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:03.399249 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:03.399261 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:03.400837 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:03.400843 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:03.401015 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:03.401050 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 10:40:03.401152 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 10:40:03.401279 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"mutex_wait_us":118,"thread_start_us":350,"threads_started":1}
I20260925 10:40:03.402626 31841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3232251de2a44dfeabae71ef64de3d0c" candidate_uuid: "dcafa1392bf049e4b508fb1ef2a74889" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d78e073456484d9082e78231eee41e48" is_pre_election: true
I20260925 10:40:03.402904 32050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:03.395077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:55185 (local address 127.23.100.3:37461)
0925 10:40:03.395243 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:03.395249 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:03.395922 (+ 673us) server_negotiation.cc:407] Connection header received
0925 10:40:03.396097 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:03.396103 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:03.396193 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:03.396275 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:40:03.396809 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:03.397737 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:03.400317 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:03.401446 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:03.401616 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:03.401697 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:03.402521 (+ 824us) server_negotiation.cc:299] Negotiation successful
0925 10:40:03.402656 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 10:40:03.402987 32086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:03.394935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.3:37461 (local address 127.23.100.1:55185)
0925 10:40:03.395794 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:03.395813 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:03.395955 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:03.396313 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:03.396317 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:03.396382 (+ 65us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:40:03.396656 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:03.396663 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:03.397888 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:03.397893 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:03.400094 (+ 2201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:03.400110 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:03.402144 (+ 2034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:03.402148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:03.402347 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:03.402387 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 10:40:03.402480 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 10:40:03.402610 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"mutex_wait_us":24,"thread_start_us":545,"threads_started":1}
I20260925 10:40:03.403090 31841 raft_consensus.cc:2454] T 3232251de2a44dfeabae71ef64de3d0c P d78e073456484d9082e78231eee41e48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcafa1392bf049e4b508fb1ef2a74889 in term 0.
I20260925 10:40:03.403523 31980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3232251de2a44dfeabae71ef64de3d0c" candidate_uuid: "dcafa1392bf049e4b508fb1ef2a74889" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a90537cc4ed423b975af8b9cd87f711" is_pre_election: true
I20260925 10:40:03.403774 31980 raft_consensus.cc:2454] T 3232251de2a44dfeabae71ef64de3d0c P 6a90537cc4ed423b975af8b9cd87f711 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcafa1392bf049e4b508fb1ef2a74889 in term 0.
I20260925 10:40:03.403760 31634 leader_election.cc:302] T 3232251de2a44dfeabae71ef64de3d0c P dcafa1392bf049e4b508fb1ef2a74889 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d78e073456484d9082e78231eee41e48, dcafa1392bf049e4b508fb1ef2a74889; no voters:
I20260925 10:40:03.404198 32062 raft_consensus.cc:2781] T 3232251de2a44dfeabae71ef64de3d0c P dcafa1392bf049e4b508fb1ef2a74889 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:40:03.404325 32062 raft_consensus.cc:482] T 3232251de2a44dfeabae71ef64de3d0c P dcafa1392bf049e4b508fb1ef2a74889 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:40:03.404438 32062 raft_consensus.cc:3037] T 3232251de2a44dfeabae71ef64de3d0c P dcafa1392bf049e4b508fb1ef2a74889 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:40:03.407605 32062 raft_consensus.cc:504] T 3232251de2a44dfeabae71ef64de3d0c P dcafa1392bf049e4b508fb1ef2a74889 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } } peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } }
I20260925 10:40:03.408278 32062 leader_election.cc:288] T 3232251de2a44dfeabae71ef64de3d0c P dcafa1392bf049e4b508fb1ef2a74889 [CANDIDATE]: Term 1 election: Requested vote from peers d78e073456484d9082e78231eee41e48 (127.23.100.2:35365), 6a90537cc4ed423b975af8b9cd87f711 (127.23.100.3:37461)
I20260925 10:40:03.408612 31841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3232251de2a44dfeabae71ef64de3d0c" candidate_uuid: "dcafa1392bf049e4b508fb1ef2a74889" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d78e073456484d9082e78231eee41e48"
I20260925 10:40:03.408804 31841 raft_consensus.cc:3037] T 3232251de2a44dfeabae71ef64de3d0c P d78e073456484d9082e78231eee41e48 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:40:03.408715 31980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3232251de2a44dfeabae71ef64de3d0c" candidate_uuid: "dcafa1392bf049e4b508fb1ef2a74889" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a90537cc4ed423b975af8b9cd87f711"
I20260925 10:40:03.408957 31980 raft_consensus.cc:3037] T 3232251de2a44dfeabae71ef64de3d0c P 6a90537cc4ed423b975af8b9cd87f711 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:40:03.411331 31980 raft_consensus.cc:2454] T 3232251de2a44dfeabae71ef64de3d0c P 6a90537cc4ed423b975af8b9cd87f711 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcafa1392bf049e4b508fb1ef2a74889 in term 1.
I20260925 10:40:03.411787 31634 leader_election.cc:302] T 3232251de2a44dfeabae71ef64de3d0c P dcafa1392bf049e4b508fb1ef2a74889 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a90537cc4ed423b975af8b9cd87f711, dcafa1392bf049e4b508fb1ef2a74889; no voters:
I20260925 10:40:03.412112 32062 raft_consensus.cc:2781] T 3232251de2a44dfeabae71ef64de3d0c P dcafa1392bf049e4b508fb1ef2a74889 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:40:03.412181 31841 raft_consensus.cc:2454] T 3232251de2a44dfeabae71ef64de3d0c P d78e073456484d9082e78231eee41e48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcafa1392bf049e4b508fb1ef2a74889 in term 1.
I20260925 10:40:03.412964 32062 raft_consensus.cc:686] T 3232251de2a44dfeabae71ef64de3d0c P dcafa1392bf049e4b508fb1ef2a74889 [term 1 LEADER]: Becoming Leader. State: Replica: dcafa1392bf049e4b508fb1ef2a74889, State: Running, Role: LEADER
I20260925 10:40:03.413324 32062 consensus_queue.cc:237] T 3232251de2a44dfeabae71ef64de3d0c P dcafa1392bf049e4b508fb1ef2a74889 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } } peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } }
I20260925 10:40:03.416745 31545 catalog_manager.cc:5495] T 3232251de2a44dfeabae71ef64de3d0c P dcafa1392bf049e4b508fb1ef2a74889 reported cstate change: term changed from 0 to 1, leader changed from <none> to dcafa1392bf049e4b508fb1ef2a74889 (127.23.100.1). New cstate: current_term: 1 leader_uuid: "dcafa1392bf049e4b508fb1ef2a74889" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } health_report { overall_health: UNKNOWN } } }
Sep 25 10:40:03 dist-test-slave-dxdd krb5kdc[31500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332800, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.62@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM
I20260925 10:40:03.466483 32095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:03.452726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48598 (local address 127.23.100.62:45071)
0925 10:40:03.453770 (+ 1044us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:03.453778 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:03.453802 (+ 24us) server_negotiation.cc:407] Connection header received
0925 10:40:03.453854 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:03.453857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:03.453903 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:03.454036 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:03.455157 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:03.455961 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:03.457112 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:03.457547 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:03.457683 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:03.461185 (+ 3502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:40:03.461190 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:40:03.461193 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:40:03.461218 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:40:03.463841 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:40:03.464493 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:40:03.464507 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:40:03.464512 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:40:03.464629 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:40:03.464991 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:40:03.464996 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:40:03.464999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:40:03.465258 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:40:03.465425 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:03.465780 (+ 355us) server_negotiation.cc:299] Negotiation successful
0925 10:40:03.466116 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":153,"server-negotiator.queue_time_us":890,"thread_start_us":749,"threads_started":1}
I20260925 10:40:03.466566 32094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:03.452680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:45071 (local address 127.0.0.1:48598)
0925 10:40:03.453502 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:03.453528 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:03.453701 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:03.454034 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:03.454039 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:03.454815 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:03.455027 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:03.455037 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:03.456095 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:03.456100 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:03.456981 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:03.456992 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:03.457697 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:03.457702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:03.457866 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:03.458519 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:03.458537 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:03.460985 (+ 2448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:40:03.463982 (+ 2997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:40:03.464002 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:40:03.464009 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:40:03.464380 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:40:03.464740 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:40:03.464745 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:40:03.464748 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:40:03.464896 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:40:03.465385 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:40:03.465392 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:40:03.465552 (+ 160us) client_negotiation.cc:769] Sending connection context
0925 10:40:03.465709 (+ 157us) client_negotiation.cc:240] Negotiation successful
0925 10:40:03.466029 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":459,"threads_started":1}
I20260925 10:40:03.471451 31545 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:48598:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 10:40:03.472808 31545 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:40:03.479797 31682 tablet_service.cc:1467] Processing CreateTablet for tablet 8eac83a3377045d1866c4a53c91c959d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6ed2425b8c234a568b0c935e7d5da90d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 10:40:03.480335 31682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8eac83a3377045d1866c4a53c91c959d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:40:03.480196 31820 tablet_service.cc:1467] Processing CreateTablet for tablet 8eac83a3377045d1866c4a53c91c959d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6ed2425b8c234a568b0c935e7d5da90d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 10:40:03.480736 31820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8eac83a3377045d1866c4a53c91c959d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:40:03.485376 32056 tablet_bootstrap.cc:492] T 8eac83a3377045d1866c4a53c91c959d P dcafa1392bf049e4b508fb1ef2a74889: Bootstrap starting.
I20260925 10:40:03.485917 31959 tablet_service.cc:1467] Processing CreateTablet for tablet 8eac83a3377045d1866c4a53c91c959d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6ed2425b8c234a568b0c935e7d5da90d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 10:40:03.486459 31959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8eac83a3377045d1866c4a53c91c959d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:40:03.487057 32056 tablet_bootstrap.cc:654] T 8eac83a3377045d1866c4a53c91c959d P dcafa1392bf049e4b508fb1ef2a74889: Neither blocks nor log segments found. Creating new log.
I20260925 10:40:03.488232 32058 tablet_bootstrap.cc:492] T 8eac83a3377045d1866c4a53c91c959d P d78e073456484d9082e78231eee41e48: Bootstrap starting.
I20260925 10:40:03.489940 32058 tablet_bootstrap.cc:654] T 8eac83a3377045d1866c4a53c91c959d P d78e073456484d9082e78231eee41e48: Neither blocks nor log segments found. Creating new log.
I20260925 10:40:03.492319 32056 tablet_bootstrap.cc:492] T 8eac83a3377045d1866c4a53c91c959d P dcafa1392bf049e4b508fb1ef2a74889: No bootstrap required, opened a new log
I20260925 10:40:03.492458 32056 ts_tablet_manager.cc:1397] T 8eac83a3377045d1866c4a53c91c959d P dcafa1392bf049e4b508fb1ef2a74889: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20260925 10:40:03.492704 32058 tablet_bootstrap.cc:492] T 8eac83a3377045d1866c4a53c91c959d P d78e073456484d9082e78231eee41e48: No bootstrap required, opened a new log
I20260925 10:40:03.492839 32058 ts_tablet_manager.cc:1397] T 8eac83a3377045d1866c4a53c91c959d P d78e073456484d9082e78231eee41e48: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260925 10:40:03.493144 32056 raft_consensus.cc:348] T 8eac83a3377045d1866c4a53c91c959d P dcafa1392bf049e4b508fb1ef2a74889 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } } peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } }
I20260925 10:40:03.493362 32056 raft_consensus.cc:374] T 8eac83a3377045d1866c4a53c91c959d P dcafa1392bf049e4b508fb1ef2a74889 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:40:03.493435 32056 raft_consensus.cc:729] T 8eac83a3377045d1866c4a53c91c959d P dcafa1392bf049e4b508fb1ef2a74889 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcafa1392bf049e4b508fb1ef2a74889, State: Initialized, Role: FOLLOWER
I20260925 10:40:03.493785 32055 tablet_bootstrap.cc:492] T 8eac83a3377045d1866c4a53c91c959d P 6a90537cc4ed423b975af8b9cd87f711: Bootstrap starting.
I20260925 10:40:03.493670 32056 consensus_queue.cc:260] T 8eac83a3377045d1866c4a53c91c959d P dcafa1392bf049e4b508fb1ef2a74889 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } } peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } }
I20260925 10:40:03.494035 32058 raft_consensus.cc:348] T 8eac83a3377045d1866c4a53c91c959d P d78e073456484d9082e78231eee41e48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } } peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } }
I20260925 10:40:03.494233 32056 ts_tablet_manager.cc:1428] T 8eac83a3377045d1866c4a53c91c959d P dcafa1392bf049e4b508fb1ef2a74889: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260925 10:40:03.494272 32058 raft_consensus.cc:374] T 8eac83a3377045d1866c4a53c91c959d P d78e073456484d9082e78231eee41e48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:40:03.494381 32058 raft_consensus.cc:729] T 8eac83a3377045d1866c4a53c91c959d P d78e073456484d9082e78231eee41e48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d78e073456484d9082e78231eee41e48, State: Initialized, Role: FOLLOWER
I20260925 10:40:03.494649 32058 consensus_queue.cc:260] T 8eac83a3377045d1866c4a53c91c959d P d78e073456484d9082e78231eee41e48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } } peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } }
I20260925 10:40:03.495087 32066 tablet_replica.cc:439] T 8eac83a3377045d1866c4a53c91c959d P d78e073456484d9082e78231eee41e48: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } } peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } } }
I20260925 10:40:03.495399 32066 tablet_replica.cc:442] T 8eac83a3377045d1866c4a53c91c959d P d78e073456484d9082e78231eee41e48: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:40:03.495460 32055 tablet_bootstrap.cc:654] T 8eac83a3377045d1866c4a53c91c959d P 6a90537cc4ed423b975af8b9cd87f711: Neither blocks nor log segments found. Creating new log.
I20260925 10:40:03.495369 32062 tablet_replica.cc:439] T 8eac83a3377045d1866c4a53c91c959d P dcafa1392bf049e4b508fb1ef2a74889: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } } peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } } }
I20260925 10:40:03.495621 32062 tablet_replica.cc:442] T 8eac83a3377045d1866c4a53c91c959d P dcafa1392bf049e4b508fb1ef2a74889: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:40:03.497457 32058 ts_tablet_manager.cc:1428] T 8eac83a3377045d1866c4a53c91c959d P d78e073456484d9082e78231eee41e48: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20260925 10:40:03.498035 32055 tablet_bootstrap.cc:492] T 8eac83a3377045d1866c4a53c91c959d P 6a90537cc4ed423b975af8b9cd87f711: No bootstrap required, opened a new log
I20260925 10:40:03.498211 32055 ts_tablet_manager.cc:1397] T 8eac83a3377045d1866c4a53c91c959d P 6a90537cc4ed423b975af8b9cd87f711: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260925 10:40:03.499022 32055 raft_consensus.cc:348] T 8eac83a3377045d1866c4a53c91c959d P 6a90537cc4ed423b975af8b9cd87f711 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } } peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } }
I20260925 10:40:03.499244 32055 raft_consensus.cc:374] T 8eac83a3377045d1866c4a53c91c959d P 6a90537cc4ed423b975af8b9cd87f711 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:40:03.499321 32055 raft_consensus.cc:729] T 8eac83a3377045d1866c4a53c91c959d P 6a90537cc4ed423b975af8b9cd87f711 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a90537cc4ed423b975af8b9cd87f711, State: Initialized, Role: FOLLOWER
I20260925 10:40:03.499558 32055 consensus_queue.cc:260] T 8eac83a3377045d1866c4a53c91c959d P 6a90537cc4ed423b975af8b9cd87f711 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } } peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } }
I20260925 10:40:03.500068 32061 tablet_replica.cc:439] T 8eac83a3377045d1866c4a53c91c959d P 6a90537cc4ed423b975af8b9cd87f711: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } } peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } } }
I20260925 10:40:03.500429 32061 tablet_replica.cc:442] T 8eac83a3377045d1866c4a53c91c959d P 6a90537cc4ed423b975af8b9cd87f711: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:40:03.500891 32055 ts_tablet_manager.cc:1428] T 8eac83a3377045d1866c4a53c91c959d P 6a90537cc4ed423b975af8b9cd87f711: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 10:40:03.589676 32066 raft_consensus.cc:482] T 8eac83a3377045d1866c4a53c91c959d P d78e073456484d9082e78231eee41e48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:40:03.589917 32066 raft_consensus.cc:504] T 8eac83a3377045d1866c4a53c91c959d P d78e073456484d9082e78231eee41e48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } } peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } }
I20260925 10:40:03.590586 32066 leader_election.cc:288] T 8eac83a3377045d1866c4a53c91c959d P d78e073456484d9082e78231eee41e48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dcafa1392bf049e4b508fb1ef2a74889 (127.23.100.1:37725), 6a90537cc4ed423b975af8b9cd87f711 (127.23.100.3:37461)
I20260925 10:40:03.591173 31702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eac83a3377045d1866c4a53c91c959d" candidate_uuid: "d78e073456484d9082e78231eee41e48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcafa1392bf049e4b508fb1ef2a74889" is_pre_election: true
I20260925 10:40:03.591184 31980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eac83a3377045d1866c4a53c91c959d" candidate_uuid: "d78e073456484d9082e78231eee41e48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a90537cc4ed423b975af8b9cd87f711" is_pre_election: true
I20260925 10:40:03.591403 31702 raft_consensus.cc:2454] T 8eac83a3377045d1866c4a53c91c959d P dcafa1392bf049e4b508fb1ef2a74889 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d78e073456484d9082e78231eee41e48 in term 0.
I20260925 10:40:03.591488 31980 raft_consensus.cc:2454] T 8eac83a3377045d1866c4a53c91c959d P 6a90537cc4ed423b975af8b9cd87f711 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d78e073456484d9082e78231eee41e48 in term 0.
I20260925 10:40:03.591949 31772 leader_election.cc:302] T 8eac83a3377045d1866c4a53c91c959d P d78e073456484d9082e78231eee41e48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d78e073456484d9082e78231eee41e48, dcafa1392bf049e4b508fb1ef2a74889; no voters:
I20260925 10:40:03.592312 32066 raft_consensus.cc:2781] T 8eac83a3377045d1866c4a53c91c959d P d78e073456484d9082e78231eee41e48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:40:03.592459 32066 raft_consensus.cc:482] T 8eac83a3377045d1866c4a53c91c959d P d78e073456484d9082e78231eee41e48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:40:03.592594 32066 raft_consensus.cc:3037] T 8eac83a3377045d1866c4a53c91c959d P d78e073456484d9082e78231eee41e48 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:40:03.595971 32066 raft_consensus.cc:504] T 8eac83a3377045d1866c4a53c91c959d P d78e073456484d9082e78231eee41e48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } } peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } }
I20260925 10:40:03.596589 32066 leader_election.cc:288] T 8eac83a3377045d1866c4a53c91c959d P d78e073456484d9082e78231eee41e48 [CANDIDATE]: Term 1 election: Requested vote from peers dcafa1392bf049e4b508fb1ef2a74889 (127.23.100.1:37725), 6a90537cc4ed423b975af8b9cd87f711 (127.23.100.3:37461)
I20260925 10:40:03.597025 31980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eac83a3377045d1866c4a53c91c959d" candidate_uuid: "d78e073456484d9082e78231eee41e48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a90537cc4ed423b975af8b9cd87f711"
I20260925 10:40:03.597019 31702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eac83a3377045d1866c4a53c91c959d" candidate_uuid: "d78e073456484d9082e78231eee41e48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcafa1392bf049e4b508fb1ef2a74889"
I20260925 10:40:03.597227 31702 raft_consensus.cc:3037] T 8eac83a3377045d1866c4a53c91c959d P dcafa1392bf049e4b508fb1ef2a74889 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:40:03.597271 31980 raft_consensus.cc:3037] T 8eac83a3377045d1866c4a53c91c959d P 6a90537cc4ed423b975af8b9cd87f711 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:40:03.599344 31980 raft_consensus.cc:2454] T 8eac83a3377045d1866c4a53c91c959d P 6a90537cc4ed423b975af8b9cd87f711 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d78e073456484d9082e78231eee41e48 in term 1.
I20260925 10:40:03.599767 31774 leader_election.cc:302] T 8eac83a3377045d1866c4a53c91c959d P d78e073456484d9082e78231eee41e48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a90537cc4ed423b975af8b9cd87f711, d78e073456484d9082e78231eee41e48; no voters:
I20260925 10:40:03.600024 32066 raft_consensus.cc:2781] T 8eac83a3377045d1866c4a53c91c959d P d78e073456484d9082e78231eee41e48 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:40:03.600221 32066 raft_consensus.cc:686] T 8eac83a3377045d1866c4a53c91c959d P d78e073456484d9082e78231eee41e48 [term 1 LEADER]: Becoming Leader. State: Replica: d78e073456484d9082e78231eee41e48, State: Running, Role: LEADER
I20260925 10:40:03.600625 31702 raft_consensus.cc:2454] T 8eac83a3377045d1866c4a53c91c959d P dcafa1392bf049e4b508fb1ef2a74889 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d78e073456484d9082e78231eee41e48 in term 1.
I20260925 10:40:03.600528 32066 consensus_queue.cc:237] T 8eac83a3377045d1866c4a53c91c959d P d78e073456484d9082e78231eee41e48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } } peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } }
I20260925 10:40:03.602425 32074 tablet_replica.cc:439] T 8eac83a3377045d1866c4a53c91c959d P d78e073456484d9082e78231eee41e48: TxnStatusTablet state changed. Reason: New leader d78e073456484d9082e78231eee41e48. Latest consensus state: current_term: 1 leader_uuid: "d78e073456484d9082e78231eee41e48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } } peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } } }
I20260925 10:40:03.602843 32074 tablet_replica.cc:442] T 8eac83a3377045d1866c4a53c91c959d P d78e073456484d9082e78231eee41e48: This TxnStatusTablet replica's current role is: LEADER
I20260925 10:40:03.603562 32100 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 10:40:03.603860 32100 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 10:40:03.604704 31545 catalog_manager.cc:5495] T 8eac83a3377045d1866c4a53c91c959d P d78e073456484d9082e78231eee41e48 reported cstate change: term changed from 0 to 1, leader changed from <none> to d78e073456484d9082e78231eee41e48 (127.23.100.2). New cstate: current_term: 1 leader_uuid: "d78e073456484d9082e78231eee41e48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } health_report { overall_health: UNKNOWN } } }
I20260925 10:40:03.629618 32054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:03.624075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33080 (local address 127.23.100.2:35365)
0925 10:40:03.624372 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:03.624376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:03.624392 (+ 16us) server_negotiation.cc:407] Connection header received
0925 10:40:03.624641 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:03.624645 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:03.624707 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:03.624782 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:40:03.625221 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:03.626035 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:03.628025 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:03.628443 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:03.628544 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:03.629031 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:40:03.629122 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:40:03.629183 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:03.629343 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 10:40:03.629400 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186}
I20260925 10:40:03.629812 32094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:03.624028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.2:35365 (local address 127.0.0.1:33080)
0925 10:40:03.624281 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:03.624302 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:03.624448 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:03.624836 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:03.624841 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:03.624854 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:40:03.625081 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:03.625092 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:03.626201 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:03.626206 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:03.627896 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:03.627911 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:03.628585 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:03.628592 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:03.628806 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:03.628882 (+ 76us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:40:03.629215 (+ 333us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:40:03.629235 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 10:40:03.629321 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 10:40:03.629484 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 10:40:03.633179 31702 raft_consensus.cc:1261] T 8eac83a3377045d1866c4a53c91c959d P dcafa1392bf049e4b508fb1ef2a74889 [term 1 FOLLOWER]: Refusing update from remote peer d78e073456484d9082e78231eee41e48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:40:03.633265 31980 raft_consensus.cc:1261] T 8eac83a3377045d1866c4a53c91c959d P 6a90537cc4ed423b975af8b9cd87f711 [term 1 FOLLOWER]: Refusing update from remote peer d78e073456484d9082e78231eee41e48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:40:03.633798 32074 consensus_queue.cc:1035] T 8eac83a3377045d1866c4a53c91c959d P d78e073456484d9082e78231eee41e48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:40:03.634135 32066 consensus_queue.cc:1035] T 8eac83a3377045d1866c4a53c91c959d P d78e073456484d9082e78231eee41e48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:40:03.636767 32062 tablet_replica.cc:439] T 8eac83a3377045d1866c4a53c91c959d P dcafa1392bf049e4b508fb1ef2a74889: TxnStatusTablet state changed. Reason: New leader d78e073456484d9082e78231eee41e48. Latest consensus state: current_term: 1 leader_uuid: "d78e073456484d9082e78231eee41e48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } } peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } } }
I20260925 10:40:03.637013 32062 tablet_replica.cc:442] T 8eac83a3377045d1866c4a53c91c959d P dcafa1392bf049e4b508fb1ef2a74889: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:40:03.637337 32061 tablet_replica.cc:439] T 8eac83a3377045d1866c4a53c91c959d P 6a90537cc4ed423b975af8b9cd87f711: TxnStatusTablet state changed. Reason: New leader d78e073456484d9082e78231eee41e48. Latest consensus state: current_term: 1 leader_uuid: "d78e073456484d9082e78231eee41e48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } } peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } } }
I20260925 10:40:03.637615 32061 tablet_replica.cc:442] T 8eac83a3377045d1866c4a53c91c959d P 6a90537cc4ed423b975af8b9cd87f711: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:40:03.639909 32063 tablet_replica.cc:439] T 8eac83a3377045d1866c4a53c91c959d P d78e073456484d9082e78231eee41e48: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d78e073456484d9082e78231eee41e48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } } peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } } }
I20260925 10:40:03.640185 32063 tablet_replica.cc:442] T 8eac83a3377045d1866c4a53c91c959d P d78e073456484d9082e78231eee41e48: This TxnStatusTablet replica's current role is: LEADER
I20260925 10:40:03.640384 32071 mvcc.cc:205] Tried to move back new op lower bound from 7333203163675975680 to 7333203163550351360. Current Snapshot: MvccSnapshot[applied={T|T < 7333203163675975680}]
I20260925 10:40:03.640821 32062 tablet_replica.cc:439] T 8eac83a3377045d1866c4a53c91c959d P dcafa1392bf049e4b508fb1ef2a74889: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d78e073456484d9082e78231eee41e48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } } peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } } }
I20260925 10:40:03.641136 32062 tablet_replica.cc:442] T 8eac83a3377045d1866c4a53c91c959d P dcafa1392bf049e4b508fb1ef2a74889: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:40:03.640993 32066 tablet_replica.cc:439] T 8eac83a3377045d1866c4a53c91c959d P d78e073456484d9082e78231eee41e48: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d78e073456484d9082e78231eee41e48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } } peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } } }
I20260925 10:40:03.641281 32066 tablet_replica.cc:442] T 8eac83a3377045d1866c4a53c91c959d P d78e073456484d9082e78231eee41e48: This TxnStatusTablet replica's current role is: LEADER
I20260925 10:40:03.643805 32070 mvcc.cc:205] Tried to move back new op lower bound from 7333203163675975680 to 7333203163550351360. Current Snapshot: MvccSnapshot[applied={T|T < 7333203163675975680 or (T in {7333203163675975680})}]
I20260925 10:40:03.644352 32061 tablet_replica.cc:439] T 8eac83a3377045d1866c4a53c91c959d P 6a90537cc4ed423b975af8b9cd87f711: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d78e073456484d9082e78231eee41e48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } } peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } } }
I20260925 10:40:03.644562 32061 tablet_replica.cc:442] T 8eac83a3377045d1866c4a53c91c959d P 6a90537cc4ed423b975af8b9cd87f711: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:40:03.652513 32104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:03.646953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51646 (local address 127.23.100.1:37725)
0925 10:40:03.647431 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:03.647436 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:03.647458 (+ 22us) server_negotiation.cc:407] Connection header received
0925 10:40:03.647614 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:03.647621 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:03.647715 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:03.647888 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:40:03.648266 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:03.649031 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:03.650380 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:03.650826 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:03.650982 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:03.651422 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:40:03.651905 (+ 483us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:40:03.652069 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:03.652125 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 10:40:03.652198 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":598,"threads_started":1}
I20260925 10:40:03.667713 32054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:03.662178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:44717 (local address 127.23.100.2:35365)
0925 10:40:03.662424 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:03.662431 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:03.662798 (+ 367us) server_negotiation.cc:407] Connection header received
0925 10:40:03.662995 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:03.662999 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:03.663065 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:03.663158 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:40:03.663628 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:03.664401 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:03.665837 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:03.666415 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:03.666547 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:03.666997 (+ 450us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:40:03.667109 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:40:03.667202 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:03.667340 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 10:40:03.667410 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 10:40:03.667796 32107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:03.662148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.2:35365 (local address 127.23.100.1:44717)
0925 10:40:03.662704 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:03.662718 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:03.662864 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:03.663176 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:03.663180 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:03.663191 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:40:03.663476 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:03.663484 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:03.664534 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:03.664539 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:03.665694 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:03.665705 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:03.666612 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:03.666618 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:03.666764 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:03.666835 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:40:03.667217 (+ 382us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:40:03.667229 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 10:40:03.667319 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 10:40:03.667441 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":316,"threads_started":1}
I20260925 10:40:03.679258 31980 raft_consensus.cc:1261] T 3232251de2a44dfeabae71ef64de3d0c P 6a90537cc4ed423b975af8b9cd87f711 [term 1 FOLLOWER]: Refusing update from remote peer dcafa1392bf049e4b508fb1ef2a74889: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:40:03.679502 31841 raft_consensus.cc:1261] T 3232251de2a44dfeabae71ef64de3d0c P d78e073456484d9082e78231eee41e48 [term 1 FOLLOWER]: Refusing update from remote peer dcafa1392bf049e4b508fb1ef2a74889: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:40:03.679937 32087 consensus_queue.cc:1035] T 3232251de2a44dfeabae71ef64de3d0c P dcafa1392bf049e4b508fb1ef2a74889 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:40:03.680634 32087 consensus_queue.cc:1035] T 3232251de2a44dfeabae71ef64de3d0c P dcafa1392bf049e4b508fb1ef2a74889 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:40:03.734441 32104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:03.728360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:50545 (local address 127.23.100.1:37725)
0925 10:40:03.728511 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:03.728515 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:03.728998 (+ 483us) server_negotiation.cc:407] Connection header received
0925 10:40:03.729209 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:03.729214 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:03.729276 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:03.729366 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:40:03.729798 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:03.730709 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:03.732507 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:03.732946 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:03.733083 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:03.733508 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:40:03.733628 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:40:03.733744 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:03.733983 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 10:40:03.734062 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 10:40:03.735255 32112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:03.728280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:37725 (local address 127.23.100.2:50545)
0925 10:40:03.728891 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:03.728914 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:03.729074 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:03.729390 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:03.729395 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:03.729408 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:40:03.729658 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:03.729666 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:03.730835 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:03.730840 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:03.732355 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:03.732368 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:03.733120 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:03.733128 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:03.733268 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:03.733317 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:40:03.733803 (+ 486us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:40:03.733822 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 10:40:03.734672 (+ 850us) client_negotiation.cc:240] Negotiation successful
0925 10:40:03.734801 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":354,"threads_started":1}
I20260925 10:40:03.780898 31543 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:40:03.781167 31543 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:40:03.784720 31543 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2620a3124bff4655ac88451061702e7d: Sending DeleteTablet for 3 replicas of tablet 3232251de2a44dfeabae71ef64de3d0c
I20260925 10:40:03.785429 31682 tablet_service.cc:1514] Processing DeleteTablet for tablet 3232251de2a44dfeabae71ef64de3d0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:40:03 UTC) from {username='oryx', principal='oryx/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:51634
I20260925 10:40:03.786197 31820 tablet_service.cc:1514] Processing DeleteTablet for tablet 3232251de2a44dfeabae71ef64de3d0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:40:03 UTC) from {username='oryx', principal='oryx/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:33060
I20260925 10:40:03.786952 31959 tablet_service.cc:1514] Processing DeleteTablet for tablet 3232251de2a44dfeabae71ef64de3d0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:40:03 UTC) from {username='oryx', principal='oryx/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:38188
I20260925 10:40:03.805029 32115 tablet_replica.cc:330] T 3232251de2a44dfeabae71ef64de3d0c P dcafa1392bf049e4b508fb1ef2a74889: stopping tablet replica
I20260925 10:40:03.810554 32115 raft_consensus.cc:2229] T 3232251de2a44dfeabae71ef64de3d0c P dcafa1392bf049e4b508fb1ef2a74889 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:40:03.811192 32115 raft_consensus.cc:2258] T 3232251de2a44dfeabae71ef64de3d0c P dcafa1392bf049e4b508fb1ef2a74889 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:40:03.812610 32115 ts_tablet_manager.cc:1905] T 3232251de2a44dfeabae71ef64de3d0c P dcafa1392bf049e4b508fb1ef2a74889: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:40:03.822820 32116 tablet_replica.cc:330] T 3232251de2a44dfeabae71ef64de3d0c P d78e073456484d9082e78231eee41e48: stopping tablet replica
I20260925 10:40:03.823036 32116 raft_consensus.cc:2229] T 3232251de2a44dfeabae71ef64de3d0c P d78e073456484d9082e78231eee41e48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:40:03.823338 32116 raft_consensus.cc:2258] T 3232251de2a44dfeabae71ef64de3d0c P d78e073456484d9082e78231eee41e48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:40:03.823809 32117 tablet_replica.cc:330] T 3232251de2a44dfeabae71ef64de3d0c P 6a90537cc4ed423b975af8b9cd87f711: stopping tablet replica
I20260925 10:40:03.824007 32117 raft_consensus.cc:2229] T 3232251de2a44dfeabae71ef64de3d0c P 6a90537cc4ed423b975af8b9cd87f711 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:40:03.824218 32117 raft_consensus.cc:2258] T 3232251de2a44dfeabae71ef64de3d0c P 6a90537cc4ed423b975af8b9cd87f711 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:40:03.825232 32117 ts_tablet_manager.cc:1905] T 3232251de2a44dfeabae71ef64de3d0c P 6a90537cc4ed423b975af8b9cd87f711: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:40:03.827409 32115 ts_tablet_manager.cc:1918] T 3232251de2a44dfeabae71ef64de3d0c P dcafa1392bf049e4b508fb1ef2a74889: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:40:03.827546 32115 log.cc:1196] T 3232251de2a44dfeabae71ef64de3d0c P dcafa1392bf049e4b508fb1ef2a74889: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-0/wal/wals/3232251de2a44dfeabae71ef64de3d0c
I20260925 10:40:03.828024 32115 ts_tablet_manager.cc:1939] T 3232251de2a44dfeabae71ef64de3d0c P dcafa1392bf049e4b508fb1ef2a74889: Deleting consensus metadata
I20260925 10:40:03.829114 31529 catalog_manager.cc:4841] TS dcafa1392bf049e4b508fb1ef2a74889 (127.23.100.1:37725): tablet 3232251de2a44dfeabae71ef64de3d0c (table test-table [id=98b803824b3142289325b194728aeeae]) successfully deleted
I20260925 10:40:03.829988 32116 ts_tablet_manager.cc:1905] T 3232251de2a44dfeabae71ef64de3d0c P d78e073456484d9082e78231eee41e48: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 25 10:40:03 dist-test-slave-dxdd krb5kdc[31500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260925 10:40:03.841070 32117 ts_tablet_manager.cc:1918] T 3232251de2a44dfeabae71ef64de3d0c P 6a90537cc4ed423b975af8b9cd87f711: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:40:03.841208 32117 log.cc:1196] T 3232251de2a44dfeabae71ef64de3d0c P 6a90537cc4ed423b975af8b9cd87f711: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-2/wal/wals/3232251de2a44dfeabae71ef64de3d0c
I20260925 10:40:03.841652 32117 ts_tablet_manager.cc:1939] T 3232251de2a44dfeabae71ef64de3d0c P 6a90537cc4ed423b975af8b9cd87f711: Deleting consensus metadata
I20260925 10:40:03.842712 31531 catalog_manager.cc:4841] TS 6a90537cc4ed423b975af8b9cd87f711 (127.23.100.3:37461): tablet 3232251de2a44dfeabae71ef64de3d0c (table test-table [id=98b803824b3142289325b194728aeeae]) successfully deleted
I20260925 10:40:03.845744 32116 ts_tablet_manager.cc:1918] T 3232251de2a44dfeabae71ef64de3d0c P d78e073456484d9082e78231eee41e48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:40:03.845892 32116 log.cc:1196] T 3232251de2a44dfeabae71ef64de3d0c P d78e073456484d9082e78231eee41e48: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-1/wal/wals/3232251de2a44dfeabae71ef64de3d0c
I20260925 10:40:03.846376 32116 ts_tablet_manager.cc:1939] T 3232251de2a44dfeabae71ef64de3d0c P d78e073456484d9082e78231eee41e48: Deleting consensus metadata
I20260925 10:40:03.848033 31531 catalog_manager.cc:4841] TS d78e073456484d9082e78231eee41e48 (127.23.100.2:35365): tablet 3232251de2a44dfeabae71ef64de3d0c (table test-table [id=98b803824b3142289325b194728aeeae]) successfully deleted
Sep 25 10:40:03 dist-test-slave-dxdd krb5kdc[31500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:40:03 dist-test-slave-dxdd krb5kdc[31500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 10:40:03.865166 32095 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:40:03.851814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48600 (local address 127.23.100.62:45071)
0925 10:40:03.852006 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:03.852013 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:03.852503 (+ 490us) server_negotiation.cc:407] Connection header received
0925 10:40:03.852692 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:03.852697 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:03.852789 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:03.852901 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:03.854560 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:03.855699 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:03.857642 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:03.858342 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:03.858475 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:03.864907 (+ 6432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48600: BlockingRecv error: recv got EOF from 127.0.0.1:48600 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 25 10:40:03 dist-test-slave-dxdd krb5kdc[31500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM
I20260925 10:40:03.887462 32095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:03.868293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48610 (local address 127.23.100.62:45071)
0925 10:40:03.868438 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:03.868454 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:03.870820 (+ 2366us) server_negotiation.cc:407] Connection header received
0925 10:40:03.870974 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:03.870979 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:03.871050 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:03.871137 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:03.872564 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:03.874045 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:03.875992 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:03.876607 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:03.876729 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:03.881247 (+ 4518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:40:03.881254 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:40:03.881259 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:40:03.881296 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:40:03.884657 (+ 3361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:40:03.885444 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:40:03.885449 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:40:03.885453 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:40:03.885615 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:40:03.886039 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:40:03.886044 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:40:03.886048 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:40:03.886384 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:40:03.886648 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:03.886911 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 10:40:03.887128 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 10:40:03.890987 31543 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:40:03.891664 31543 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:40:03.899206 31682 tablet_service.cc:1467] Processing CreateTablet for tablet 726046a45c7e4acaa1336fdd37651a2a (DEFAULT_TABLE table=test-table [id=42052bf9eca84e439b87a9016b6ab774]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:40:03.899292 31820 tablet_service.cc:1467] Processing CreateTablet for tablet 726046a45c7e4acaa1336fdd37651a2a (DEFAULT_TABLE table=test-table [id=42052bf9eca84e439b87a9016b6ab774]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:40:03.899631 31682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 726046a45c7e4acaa1336fdd37651a2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:40:03.899757 31820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 726046a45c7e4acaa1336fdd37651a2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:40:03.902909 31959 tablet_service.cc:1467] Processing CreateTablet for tablet 726046a45c7e4acaa1336fdd37651a2a (DEFAULT_TABLE table=test-table [id=42052bf9eca84e439b87a9016b6ab774]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:40:03.903363 31959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 726046a45c7e4acaa1336fdd37651a2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:40:03.908293 32058 tablet_bootstrap.cc:492] T 726046a45c7e4acaa1336fdd37651a2a P d78e073456484d9082e78231eee41e48: Bootstrap starting.
I20260925 10:40:03.909876 32058 tablet_bootstrap.cc:654] T 726046a45c7e4acaa1336fdd37651a2a P d78e073456484d9082e78231eee41e48: Neither blocks nor log segments found. Creating new log.
I20260925 10:40:03.912657 32055 tablet_bootstrap.cc:492] T 726046a45c7e4acaa1336fdd37651a2a P 6a90537cc4ed423b975af8b9cd87f711: Bootstrap starting.
I20260925 10:40:03.914186 32055 tablet_bootstrap.cc:654] T 726046a45c7e4acaa1336fdd37651a2a P 6a90537cc4ed423b975af8b9cd87f711: Neither blocks nor log segments found. Creating new log.
I20260925 10:40:03.918816 32056 tablet_bootstrap.cc:492] T 726046a45c7e4acaa1336fdd37651a2a P dcafa1392bf049e4b508fb1ef2a74889: Bootstrap starting.
I20260925 10:40:03.920532 32056 tablet_bootstrap.cc:654] T 726046a45c7e4acaa1336fdd37651a2a P dcafa1392bf049e4b508fb1ef2a74889: Neither blocks nor log segments found. Creating new log.
I20260925 10:40:03.922497 32058 tablet_bootstrap.cc:492] T 726046a45c7e4acaa1336fdd37651a2a P d78e073456484d9082e78231eee41e48: No bootstrap required, opened a new log
I20260925 10:40:03.922712 32058 ts_tablet_manager.cc:1397] T 726046a45c7e4acaa1336fdd37651a2a P d78e073456484d9082e78231eee41e48: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.004s
I20260925 10:40:03.924582 32058 raft_consensus.cc:348] T 726046a45c7e4acaa1336fdd37651a2a P d78e073456484d9082e78231eee41e48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } } peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } }
I20260925 10:40:03.924907 32058 raft_consensus.cc:374] T 726046a45c7e4acaa1336fdd37651a2a P d78e073456484d9082e78231eee41e48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:40:03.925040 32058 raft_consensus.cc:729] T 726046a45c7e4acaa1336fdd37651a2a P d78e073456484d9082e78231eee41e48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d78e073456484d9082e78231eee41e48, State: Initialized, Role: FOLLOWER
I20260925 10:40:03.925282 32058 consensus_queue.cc:260] T 726046a45c7e4acaa1336fdd37651a2a P d78e073456484d9082e78231eee41e48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } } peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } }
I20260925 10:40:03.928202 32058 ts_tablet_manager.cc:1428] T 726046a45c7e4acaa1336fdd37651a2a P d78e073456484d9082e78231eee41e48: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260925 10:40:03.935771 32056 tablet_bootstrap.cc:492] T 726046a45c7e4acaa1336fdd37651a2a P dcafa1392bf049e4b508fb1ef2a74889: No bootstrap required, opened a new log
I20260925 10:40:03.935935 32056 ts_tablet_manager.cc:1397] T 726046a45c7e4acaa1336fdd37651a2a P dcafa1392bf049e4b508fb1ef2a74889: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.007s
I20260925 10:40:03.936700 32056 raft_consensus.cc:348] T 726046a45c7e4acaa1336fdd37651a2a P dcafa1392bf049e4b508fb1ef2a74889 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } } peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } }
I20260925 10:40:03.936968 32056 raft_consensus.cc:374] T 726046a45c7e4acaa1336fdd37651a2a P dcafa1392bf049e4b508fb1ef2a74889 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:40:03.937062 32056 raft_consensus.cc:729] T 726046a45c7e4acaa1336fdd37651a2a P dcafa1392bf049e4b508fb1ef2a74889 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcafa1392bf049e4b508fb1ef2a74889, State: Initialized, Role: FOLLOWER
I20260925 10:40:03.937305 32056 consensus_queue.cc:260] T 726046a45c7e4acaa1336fdd37651a2a P dcafa1392bf049e4b508fb1ef2a74889 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } } peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } }
I20260925 10:40:03.937987 32055 tablet_bootstrap.cc:492] T 726046a45c7e4acaa1336fdd37651a2a P 6a90537cc4ed423b975af8b9cd87f711: No bootstrap required, opened a new log
I20260925 10:40:03.938201 32055 ts_tablet_manager.cc:1397] T 726046a45c7e4acaa1336fdd37651a2a P 6a90537cc4ed423b975af8b9cd87f711: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.005s
I20260925 10:40:03.939015 32055 raft_consensus.cc:348] T 726046a45c7e4acaa1336fdd37651a2a P 6a90537cc4ed423b975af8b9cd87f711 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } } peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } }
I20260925 10:40:03.939311 32055 raft_consensus.cc:374] T 726046a45c7e4acaa1336fdd37651a2a P 6a90537cc4ed423b975af8b9cd87f711 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:40:03.939437 32055 raft_consensus.cc:729] T 726046a45c7e4acaa1336fdd37651a2a P 6a90537cc4ed423b975af8b9cd87f711 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a90537cc4ed423b975af8b9cd87f711, State: Initialized, Role: FOLLOWER
I20260925 10:40:03.939968 32056 ts_tablet_manager.cc:1428] T 726046a45c7e4acaa1336fdd37651a2a P dcafa1392bf049e4b508fb1ef2a74889: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260925 10:40:03.939857 32055 consensus_queue.cc:260] T 726046a45c7e4acaa1336fdd37651a2a P 6a90537cc4ed423b975af8b9cd87f711 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } } peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } }
I20260925 10:40:03.941250 32055 ts_tablet_manager.cc:1428] T 726046a45c7e4acaa1336fdd37651a2a P 6a90537cc4ed423b975af8b9cd87f711: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 10:40:04.127020 32074 raft_consensus.cc:482] T 726046a45c7e4acaa1336fdd37651a2a P d78e073456484d9082e78231eee41e48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:40:04.127300 32074 raft_consensus.cc:504] T 726046a45c7e4acaa1336fdd37651a2a P d78e073456484d9082e78231eee41e48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } } peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } }
I20260925 10:40:04.128085 32074 leader_election.cc:288] T 726046a45c7e4acaa1336fdd37651a2a P d78e073456484d9082e78231eee41e48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dcafa1392bf049e4b508fb1ef2a74889 (127.23.100.1:37725), 6a90537cc4ed423b975af8b9cd87f711 (127.23.100.3:37461)
I20260925 10:40:04.128684 31702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "726046a45c7e4acaa1336fdd37651a2a" candidate_uuid: "d78e073456484d9082e78231eee41e48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcafa1392bf049e4b508fb1ef2a74889" is_pre_election: true
I20260925 10:40:04.128845 31980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "726046a45c7e4acaa1336fdd37651a2a" candidate_uuid: "d78e073456484d9082e78231eee41e48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a90537cc4ed423b975af8b9cd87f711" is_pre_election: true
I20260925 10:40:04.128960 31702 raft_consensus.cc:2454] T 726046a45c7e4acaa1336fdd37651a2a P dcafa1392bf049e4b508fb1ef2a74889 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d78e073456484d9082e78231eee41e48 in term 0.
I20260925 10:40:04.129118 31980 raft_consensus.cc:2454] T 726046a45c7e4acaa1336fdd37651a2a P 6a90537cc4ed423b975af8b9cd87f711 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d78e073456484d9082e78231eee41e48 in term 0.
I20260925 10:40:04.129647 31774 leader_election.cc:302] T 726046a45c7e4acaa1336fdd37651a2a P d78e073456484d9082e78231eee41e48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a90537cc4ed423b975af8b9cd87f711, d78e073456484d9082e78231eee41e48; no voters:
I20260925 10:40:04.129953 32074 raft_consensus.cc:2781] T 726046a45c7e4acaa1336fdd37651a2a P d78e073456484d9082e78231eee41e48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:40:04.130059 32074 raft_consensus.cc:482] T 726046a45c7e4acaa1336fdd37651a2a P d78e073456484d9082e78231eee41e48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:40:04.130167 32074 raft_consensus.cc:3037] T 726046a45c7e4acaa1336fdd37651a2a P d78e073456484d9082e78231eee41e48 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:40:04.133330 32074 raft_consensus.cc:504] T 726046a45c7e4acaa1336fdd37651a2a P d78e073456484d9082e78231eee41e48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } } peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } }
I20260925 10:40:04.133958 32074 leader_election.cc:288] T 726046a45c7e4acaa1336fdd37651a2a P d78e073456484d9082e78231eee41e48 [CANDIDATE]: Term 1 election: Requested vote from peers dcafa1392bf049e4b508fb1ef2a74889 (127.23.100.1:37725), 6a90537cc4ed423b975af8b9cd87f711 (127.23.100.3:37461)
I20260925 10:40:04.134538 31702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "726046a45c7e4acaa1336fdd37651a2a" candidate_uuid: "d78e073456484d9082e78231eee41e48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcafa1392bf049e4b508fb1ef2a74889"
I20260925 10:40:04.134749 31702 raft_consensus.cc:3037] T 726046a45c7e4acaa1336fdd37651a2a P dcafa1392bf049e4b508fb1ef2a74889 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:40:04.134922 31980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "726046a45c7e4acaa1336fdd37651a2a" candidate_uuid: "d78e073456484d9082e78231eee41e48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a90537cc4ed423b975af8b9cd87f711"
I20260925 10:40:04.135190 31980 raft_consensus.cc:3037] T 726046a45c7e4acaa1336fdd37651a2a P 6a90537cc4ed423b975af8b9cd87f711 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:40:04.138093 31702 raft_consensus.cc:2454] T 726046a45c7e4acaa1336fdd37651a2a P dcafa1392bf049e4b508fb1ef2a74889 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d78e073456484d9082e78231eee41e48 in term 1.
I20260925 10:40:04.138209 31980 raft_consensus.cc:2454] T 726046a45c7e4acaa1336fdd37651a2a P 6a90537cc4ed423b975af8b9cd87f711 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d78e073456484d9082e78231eee41e48 in term 1.
I20260925 10:40:04.138696 31772 leader_election.cc:302] T 726046a45c7e4acaa1336fdd37651a2a P d78e073456484d9082e78231eee41e48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d78e073456484d9082e78231eee41e48, dcafa1392bf049e4b508fb1ef2a74889; no voters:
I20260925 10:40:04.139079 32074 raft_consensus.cc:2781] T 726046a45c7e4acaa1336fdd37651a2a P d78e073456484d9082e78231eee41e48 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:40:04.139379 32074 raft_consensus.cc:686] T 726046a45c7e4acaa1336fdd37651a2a P d78e073456484d9082e78231eee41e48 [term 1 LEADER]: Becoming Leader. State: Replica: d78e073456484d9082e78231eee41e48, State: Running, Role: LEADER
I20260925 10:40:04.139662 32074 consensus_queue.cc:237] T 726046a45c7e4acaa1336fdd37651a2a P d78e073456484d9082e78231eee41e48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } } peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } }
I20260925 10:40:04.142850 31543 catalog_manager.cc:5495] T 726046a45c7e4acaa1336fdd37651a2a P d78e073456484d9082e78231eee41e48 reported cstate change: term changed from 0 to 1, leader changed from <none> to d78e073456484d9082e78231eee41e48 (127.23.100.2). New cstate: current_term: 1 leader_uuid: "d78e073456484d9082e78231eee41e48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } health_report { overall_health: UNKNOWN } } }
I20260925 10:40:04.214151 32135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:04.207518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33084 (local address 127.23.100.2:35365)
0925 10:40:04.207962 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:04.207968 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:04.207985 (+ 17us) server_negotiation.cc:407] Connection header received
0925 10:40:04.208047 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:04.208052 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:04.208127 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:04.208231 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:40:04.208671 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:04.209732 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:04.211889 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:04.212537 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:04.212658 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:04.213206 (+ 548us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:40:04.213330 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:40:04.213416 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:04.213637 (+ 221us) server_negotiation.cc:299] Negotiation successful
0925 10:40:04.213712 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":208,"threads_started":1}
I20260925 10:40:04.216872 31702 raft_consensus.cc:1261] T 726046a45c7e4acaa1336fdd37651a2a P dcafa1392bf049e4b508fb1ef2a74889 [term 1 FOLLOWER]: Refusing update from remote peer d78e073456484d9082e78231eee41e48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:40:04.216969 31980 raft_consensus.cc:1261] T 726046a45c7e4acaa1336fdd37651a2a P 6a90537cc4ed423b975af8b9cd87f711 [term 1 FOLLOWER]: Refusing update from remote peer d78e073456484d9082e78231eee41e48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:40:04.217556 32066 consensus_queue.cc:1035] T 726046a45c7e4acaa1336fdd37651a2a P d78e073456484d9082e78231eee41e48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:40:04.217922 32074 consensus_queue.cc:1035] T 726046a45c7e4acaa1336fdd37651a2a P d78e073456484d9082e78231eee41e48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:40:04.238933 31543 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:40:04.239148 31543 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:40:04.242205 31543 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2620a3124bff4655ac88451061702e7d: Sending DeleteTablet for 3 replicas of tablet 726046a45c7e4acaa1336fdd37651a2a
I20260925 10:40:04.243144 31682 tablet_service.cc:1514] Processing DeleteTablet for tablet 726046a45c7e4acaa1336fdd37651a2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:40:04 UTC) from {username='oryx', principal='oryx/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:51634
I20260925 10:40:04.243397 31820 tablet_service.cc:1514] Processing DeleteTablet for tablet 726046a45c7e4acaa1336fdd37651a2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:40:04 UTC) from {username='oryx', principal='oryx/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:33060
I20260925 10:40:04.243474 32115 tablet_replica.cc:330] T 726046a45c7e4acaa1336fdd37651a2a P dcafa1392bf049e4b508fb1ef2a74889: stopping tablet replica
I20260925 10:40:04.243624 32116 tablet_replica.cc:330] T 726046a45c7e4acaa1336fdd37651a2a P d78e073456484d9082e78231eee41e48: stopping tablet replica
I20260925 10:40:04.243709 32115 raft_consensus.cc:2229] T 726046a45c7e4acaa1336fdd37651a2a P dcafa1392bf049e4b508fb1ef2a74889 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:40:04.243814 32116 raft_consensus.cc:2229] T 726046a45c7e4acaa1336fdd37651a2a P d78e073456484d9082e78231eee41e48 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:40:04.243978 32115 raft_consensus.cc:2258] T 726046a45c7e4acaa1336fdd37651a2a P dcafa1392bf049e4b508fb1ef2a74889 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:40:04.244091 32116 raft_consensus.cc:2258] T 726046a45c7e4acaa1336fdd37651a2a P d78e073456484d9082e78231eee41e48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:40:04.244289 31681 tablet_service.cc:1514] Processing DeleteTablet for tablet 726046a45c7e4acaa1336fdd37651a2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:40:04 UTC) from {username='oryx', principal='oryx/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:51634
I20260925 10:40:04.244738 31959 tablet_service.cc:1514] Processing DeleteTablet for tablet 726046a45c7e4acaa1336fdd37651a2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:40:04 UTC) from {username='oryx', principal='oryx/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:38188
I20260925 10:40:04.245097 32116 ts_tablet_manager.cc:1905] T 726046a45c7e4acaa1336fdd37651a2a P d78e073456484d9082e78231eee41e48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:40:04.245831 32115 ts_tablet_manager.cc:1905] T 726046a45c7e4acaa1336fdd37651a2a P dcafa1392bf049e4b508fb1ef2a74889: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:40:04.245764 31543 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:40:04.246619 31543 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:40:04.248445 32117 tablet_replica.cc:330] T 726046a45c7e4acaa1336fdd37651a2a P 6a90537cc4ed423b975af8b9cd87f711: stopping tablet replica
I20260925 10:40:04.248651 32117 raft_consensus.cc:2229] T 726046a45c7e4acaa1336fdd37651a2a P 6a90537cc4ed423b975af8b9cd87f711 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:40:04.248884 32117 raft_consensus.cc:2258] T 726046a45c7e4acaa1336fdd37651a2a P 6a90537cc4ed423b975af8b9cd87f711 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:40:04.249879 32117 ts_tablet_manager.cc:1905] T 726046a45c7e4acaa1336fdd37651a2a P 6a90537cc4ed423b975af8b9cd87f711: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:40:04.251894 32115 ts_tablet_manager.cc:1918] T 726046a45c7e4acaa1336fdd37651a2a P dcafa1392bf049e4b508fb1ef2a74889: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:40:04.252045 32115 log.cc:1196] T 726046a45c7e4acaa1336fdd37651a2a P dcafa1392bf049e4b508fb1ef2a74889: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-0/wal/wals/726046a45c7e4acaa1336fdd37651a2a
I20260925 10:40:04.252481 32115 ts_tablet_manager.cc:1939] T 726046a45c7e4acaa1336fdd37651a2a P dcafa1392bf049e4b508fb1ef2a74889: Deleting consensus metadata
I20260925 10:40:04.252966 32116 ts_tablet_manager.cc:1918] T 726046a45c7e4acaa1336fdd37651a2a P d78e073456484d9082e78231eee41e48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:40:04.253115 32116 log.cc:1196] T 726046a45c7e4acaa1336fdd37651a2a P d78e073456484d9082e78231eee41e48: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-1/wal/wals/726046a45c7e4acaa1336fdd37651a2a
I20260925 10:40:04.253508 32116 ts_tablet_manager.cc:1939] T 726046a45c7e4acaa1336fdd37651a2a P d78e073456484d9082e78231eee41e48: Deleting consensus metadata
I20260925 10:40:04.254536 31529 catalog_manager.cc:4841] TS dcafa1392bf049e4b508fb1ef2a74889 (127.23.100.1:37725): tablet 726046a45c7e4acaa1336fdd37651a2a (table test-table [id=42052bf9eca84e439b87a9016b6ab774]) successfully deleted
I20260925 10:40:04.254807 31959 tablet_service.cc:1467] Processing CreateTablet for tablet e4304ab12eaa4114908d192ef6fbb573 (DEFAULT_TABLE table=test-table [id=758953fa9b234fad8d2e4cba93141c9e]), partition=RANGE (key) PARTITION UNBOUNDED
W20260925 10:40:04.254988 31529 catalog_manager.cc:4808] TS dcafa1392bf049e4b508fb1ef2a74889 (127.23.100.1:37725): delete failed for tablet 726046a45c7e4acaa1336fdd37651a2a because the tablet was not found. No further retry: Not found: Tablet not found: 726046a45c7e4acaa1336fdd37651a2a
I20260925 10:40:04.255209 31959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4304ab12eaa4114908d192ef6fbb573. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:40:04.255358 31820 tablet_service.cc:1467] Processing CreateTablet for tablet e4304ab12eaa4114908d192ef6fbb573 (DEFAULT_TABLE table=test-table [id=758953fa9b234fad8d2e4cba93141c9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:40:04.255523 31681 tablet_service.cc:1467] Processing CreateTablet for tablet e4304ab12eaa4114908d192ef6fbb573 (DEFAULT_TABLE table=test-table [id=758953fa9b234fad8d2e4cba93141c9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:40:04.255698 31820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4304ab12eaa4114908d192ef6fbb573. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:40:04.255896 31681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4304ab12eaa4114908d192ef6fbb573. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:40:04.256495 32117 ts_tablet_manager.cc:1918] T 726046a45c7e4acaa1336fdd37651a2a P 6a90537cc4ed423b975af8b9cd87f711: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:40:04.256644 32117 log.cc:1196] T 726046a45c7e4acaa1336fdd37651a2a P 6a90537cc4ed423b975af8b9cd87f711: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-2/wal/wals/726046a45c7e4acaa1336fdd37651a2a
I20260925 10:40:04.257081 32117 ts_tablet_manager.cc:1939] T 726046a45c7e4acaa1336fdd37651a2a P 6a90537cc4ed423b975af8b9cd87f711: Deleting consensus metadata
I20260925 10:40:04.258947 31531 catalog_manager.cc:4841] TS 6a90537cc4ed423b975af8b9cd87f711 (127.23.100.3:37461): tablet 726046a45c7e4acaa1336fdd37651a2a (table test-table [id=42052bf9eca84e439b87a9016b6ab774]) successfully deleted
I20260925 10:40:04.259146 31531 catalog_manager.cc:4841] TS d78e073456484d9082e78231eee41e48 (127.23.100.2:35365): tablet 726046a45c7e4acaa1336fdd37651a2a (table test-table [id=42052bf9eca84e439b87a9016b6ab774]) successfully deleted
I20260925 10:40:04.260855 32056 tablet_bootstrap.cc:492] T e4304ab12eaa4114908d192ef6fbb573 P dcafa1392bf049e4b508fb1ef2a74889: Bootstrap starting.
I20260925 10:40:04.261250 32055 tablet_bootstrap.cc:492] T e4304ab12eaa4114908d192ef6fbb573 P 6a90537cc4ed423b975af8b9cd87f711: Bootstrap starting.
I20260925 10:40:04.262523 32056 tablet_bootstrap.cc:654] T e4304ab12eaa4114908d192ef6fbb573 P dcafa1392bf049e4b508fb1ef2a74889: Neither blocks nor log segments found. Creating new log.
I20260925 10:40:04.262794 32055 tablet_bootstrap.cc:654] T e4304ab12eaa4114908d192ef6fbb573 P 6a90537cc4ed423b975af8b9cd87f711: Neither blocks nor log segments found. Creating new log.
I20260925 10:40:04.263957 32058 tablet_bootstrap.cc:492] T e4304ab12eaa4114908d192ef6fbb573 P d78e073456484d9082e78231eee41e48: Bootstrap starting.
I20260925 10:40:04.265223 32055 tablet_bootstrap.cc:492] T e4304ab12eaa4114908d192ef6fbb573 P 6a90537cc4ed423b975af8b9cd87f711: No bootstrap required, opened a new log
I20260925 10:40:04.265370 32055 ts_tablet_manager.cc:1397] T e4304ab12eaa4114908d192ef6fbb573 P 6a90537cc4ed423b975af8b9cd87f711: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260925 10:40:04.265545 32056 tablet_bootstrap.cc:492] T e4304ab12eaa4114908d192ef6fbb573 P dcafa1392bf049e4b508fb1ef2a74889: No bootstrap required, opened a new log
I20260925 10:40:04.265697 32056 ts_tablet_manager.cc:1397] T e4304ab12eaa4114908d192ef6fbb573 P dcafa1392bf049e4b508fb1ef2a74889: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260925 10:40:04.265736 32058 tablet_bootstrap.cc:654] T e4304ab12eaa4114908d192ef6fbb573 P d78e073456484d9082e78231eee41e48: Neither blocks nor log segments found. Creating new log.
I20260925 10:40:04.266144 32055 raft_consensus.cc:348] T e4304ab12eaa4114908d192ef6fbb573 P 6a90537cc4ed423b975af8b9cd87f711 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } } peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } }
I20260925 10:40:04.266412 32056 raft_consensus.cc:348] T e4304ab12eaa4114908d192ef6fbb573 P dcafa1392bf049e4b508fb1ef2a74889 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } } peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } }
I20260925 10:40:04.266597 32055 raft_consensus.cc:374] T e4304ab12eaa4114908d192ef6fbb573 P 6a90537cc4ed423b975af8b9cd87f711 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:40:04.266642 32056 raft_consensus.cc:374] T e4304ab12eaa4114908d192ef6fbb573 P dcafa1392bf049e4b508fb1ef2a74889 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:40:04.266688 32055 raft_consensus.cc:729] T e4304ab12eaa4114908d192ef6fbb573 P 6a90537cc4ed423b975af8b9cd87f711 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a90537cc4ed423b975af8b9cd87f711, State: Initialized, Role: FOLLOWER
I20260925 10:40:04.266716 32056 raft_consensus.cc:729] T e4304ab12eaa4114908d192ef6fbb573 P dcafa1392bf049e4b508fb1ef2a74889 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcafa1392bf049e4b508fb1ef2a74889, State: Initialized, Role: FOLLOWER
I20260925 10:40:04.266954 32055 consensus_queue.cc:260] T e4304ab12eaa4114908d192ef6fbb573 P 6a90537cc4ed423b975af8b9cd87f711 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } } peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } }
I20260925 10:40:04.267264 32056 consensus_queue.cc:260] T e4304ab12eaa4114908d192ef6fbb573 P dcafa1392bf049e4b508fb1ef2a74889 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } } peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } }
I20260925 10:40:04.267563 32055 ts_tablet_manager.cc:1428] T e4304ab12eaa4114908d192ef6fbb573 P 6a90537cc4ed423b975af8b9cd87f711: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:40:04.267865 32056 ts_tablet_manager.cc:1428] T e4304ab12eaa4114908d192ef6fbb573 P dcafa1392bf049e4b508fb1ef2a74889: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:40:04.269805 32058 tablet_bootstrap.cc:492] T e4304ab12eaa4114908d192ef6fbb573 P d78e073456484d9082e78231eee41e48: No bootstrap required, opened a new log
I20260925 10:40:04.269958 32058 ts_tablet_manager.cc:1397] T e4304ab12eaa4114908d192ef6fbb573 P d78e073456484d9082e78231eee41e48: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 10:40:04.270668 32058 raft_consensus.cc:348] T e4304ab12eaa4114908d192ef6fbb573 P d78e073456484d9082e78231eee41e48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } } peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } }
I20260925 10:40:04.270872 32058 raft_consensus.cc:374] T e4304ab12eaa4114908d192ef6fbb573 P d78e073456484d9082e78231eee41e48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:40:04.270951 32058 raft_consensus.cc:729] T e4304ab12eaa4114908d192ef6fbb573 P d78e073456484d9082e78231eee41e48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d78e073456484d9082e78231eee41e48, State: Initialized, Role: FOLLOWER
I20260925 10:40:04.271203 32058 consensus_queue.cc:260] T e4304ab12eaa4114908d192ef6fbb573 P d78e073456484d9082e78231eee41e48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } } peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } }
I20260925 10:40:04.271737 32058 ts_tablet_manager.cc:1428] T e4304ab12eaa4114908d192ef6fbb573 P d78e073456484d9082e78231eee41e48: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260925 10:40:04.311364 32061 raft_consensus.cc:482] T e4304ab12eaa4114908d192ef6fbb573 P 6a90537cc4ed423b975af8b9cd87f711 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:40:04.311548 32061 raft_consensus.cc:504] T e4304ab12eaa4114908d192ef6fbb573 P 6a90537cc4ed423b975af8b9cd87f711 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } } peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } }
I20260925 10:40:04.313113 32061 leader_election.cc:288] T e4304ab12eaa4114908d192ef6fbb573 P 6a90537cc4ed423b975af8b9cd87f711 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dcafa1392bf049e4b508fb1ef2a74889 (127.23.100.1:37725), d78e073456484d9082e78231eee41e48 (127.23.100.2:35365)
I20260925 10:40:04.319708 32143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:04.313355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:33093 (local address 127.23.100.1:37725)
0925 10:40:04.314267 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:04.314273 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:04.314630 (+ 357us) server_negotiation.cc:407] Connection header received
0925 10:40:04.314909 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:04.314915 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:04.314994 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:04.315115 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:40:04.315546 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:04.316262 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:04.317905 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:04.318745 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:04.318907 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:04.318980 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:04.319221 (+ 241us) server_negotiation.cc:299] Negotiation successful
0925 10:40:04.319387 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":773,"thread_start_us":584,"threads_started":1}
I20260925 10:40:04.319726 32142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:04.313283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:37725 (local address 127.23.100.3:33093)
0925 10:40:04.314521 (+ 1238us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:04.314539 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:04.314745 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:04.315124 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:04.315129 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:04.315154 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:40:04.315393 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:04.315401 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:04.316398 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:04.316404 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:04.317751 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:04.317762 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:04.318892 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:04.318897 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:04.319069 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:04.319092 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 10:40:04.319188 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 10:40:04.319308 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1055,"thread_start_us":875,"threads_started":1}
I20260925 10:40:04.320072 32135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:04.313460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:49911 (local address 127.23.100.2:35365)
0925 10:40:04.313719 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:04.313728 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:04.314927 (+ 1199us) server_negotiation.cc:407] Connection header received
0925 10:40:04.315167 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:04.315175 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:04.315246 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:04.315339 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:40:04.315899 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:04.316592 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:04.318365 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:04.319125 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:04.319290 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:04.319364 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:04.319688 (+ 324us) server_negotiation.cc:299] Negotiation successful
0925 10:40:04.319807 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 10:40:04.320111 32144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:04.313376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.2:35365 (local address 127.23.100.3:49911)
0925 10:40:04.314769 (+ 1393us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:04.314787 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:04.315026 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:04.315367 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:04.315371 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:04.315400 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:40:04.315739 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:04.315746 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:04.316727 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:04.316734 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:04.318209 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:04.318220 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:04.319336 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:04.319342 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:04.319492 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:04.319541 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 10:40:04.319657 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 10:40:04.319780 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1246,"thread_start_us":1035,"threads_started":1}
I20260925 10:40:04.320657 31702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4304ab12eaa4114908d192ef6fbb573" candidate_uuid: "6a90537cc4ed423b975af8b9cd87f711" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcafa1392bf049e4b508fb1ef2a74889" is_pre_election: true
I20260925 10:40:04.320784 31840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4304ab12eaa4114908d192ef6fbb573" candidate_uuid: "6a90537cc4ed423b975af8b9cd87f711" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d78e073456484d9082e78231eee41e48" is_pre_election: true
I20260925 10:40:04.320884 31702 raft_consensus.cc:2454] T e4304ab12eaa4114908d192ef6fbb573 P dcafa1392bf049e4b508fb1ef2a74889 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a90537cc4ed423b975af8b9cd87f711 in term 0.
I20260925 10:40:04.321029 31840 raft_consensus.cc:2454] T e4304ab12eaa4114908d192ef6fbb573 P d78e073456484d9082e78231eee41e48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a90537cc4ed423b975af8b9cd87f711 in term 0.
I20260925 10:40:04.321381 31911 leader_election.cc:302] T e4304ab12eaa4114908d192ef6fbb573 P 6a90537cc4ed423b975af8b9cd87f711 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a90537cc4ed423b975af8b9cd87f711, dcafa1392bf049e4b508fb1ef2a74889; no voters:
I20260925 10:40:04.321810 32061 raft_consensus.cc:2781] T e4304ab12eaa4114908d192ef6fbb573 P 6a90537cc4ed423b975af8b9cd87f711 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:40:04.321941 32061 raft_consensus.cc:482] T e4304ab12eaa4114908d192ef6fbb573 P 6a90537cc4ed423b975af8b9cd87f711 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:40:04.322057 32061 raft_consensus.cc:3037] T e4304ab12eaa4114908d192ef6fbb573 P 6a90537cc4ed423b975af8b9cd87f711 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:40:04.324287 32061 raft_consensus.cc:504] T e4304ab12eaa4114908d192ef6fbb573 P 6a90537cc4ed423b975af8b9cd87f711 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } } peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } }
I20260925 10:40:04.324851 32061 leader_election.cc:288] T e4304ab12eaa4114908d192ef6fbb573 P 6a90537cc4ed423b975af8b9cd87f711 [CANDIDATE]: Term 1 election: Requested vote from peers dcafa1392bf049e4b508fb1ef2a74889 (127.23.100.1:37725), d78e073456484d9082e78231eee41e48 (127.23.100.2:35365)
I20260925 10:40:04.325266 31702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4304ab12eaa4114908d192ef6fbb573" candidate_uuid: "6a90537cc4ed423b975af8b9cd87f711" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcafa1392bf049e4b508fb1ef2a74889"
I20260925 10:40:04.325469 31840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4304ab12eaa4114908d192ef6fbb573" candidate_uuid: "6a90537cc4ed423b975af8b9cd87f711" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d78e073456484d9082e78231eee41e48"
I20260925 10:40:04.325531 31702 raft_consensus.cc:3037] T e4304ab12eaa4114908d192ef6fbb573 P dcafa1392bf049e4b508fb1ef2a74889 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:40:04.325680 31840 raft_consensus.cc:3037] T e4304ab12eaa4114908d192ef6fbb573 P d78e073456484d9082e78231eee41e48 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:40:04.328099 31702 raft_consensus.cc:2454] T e4304ab12eaa4114908d192ef6fbb573 P dcafa1392bf049e4b508fb1ef2a74889 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a90537cc4ed423b975af8b9cd87f711 in term 1.
I20260925 10:40:04.328226 31840 raft_consensus.cc:2454] T e4304ab12eaa4114908d192ef6fbb573 P d78e073456484d9082e78231eee41e48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a90537cc4ed423b975af8b9cd87f711 in term 1.
I20260925 10:40:04.328439 31911 leader_election.cc:302] T e4304ab12eaa4114908d192ef6fbb573 P 6a90537cc4ed423b975af8b9cd87f711 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a90537cc4ed423b975af8b9cd87f711, dcafa1392bf049e4b508fb1ef2a74889; no voters:
I20260925 10:40:04.328719 32061 raft_consensus.cc:2781] T e4304ab12eaa4114908d192ef6fbb573 P 6a90537cc4ed423b975af8b9cd87f711 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:40:04.329138 32061 raft_consensus.cc:686] T e4304ab12eaa4114908d192ef6fbb573 P 6a90537cc4ed423b975af8b9cd87f711 [term 1 LEADER]: Becoming Leader. State: Replica: 6a90537cc4ed423b975af8b9cd87f711, State: Running, Role: LEADER
I20260925 10:40:04.329504 32061 consensus_queue.cc:237] T e4304ab12eaa4114908d192ef6fbb573 P 6a90537cc4ed423b975af8b9cd87f711 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } } peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } }
I20260925 10:40:04.332458 31543 catalog_manager.cc:5495] T e4304ab12eaa4114908d192ef6fbb573 P 6a90537cc4ed423b975af8b9cd87f711 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a90537cc4ed423b975af8b9cd87f711 (127.23.100.3). New cstate: current_term: 1 leader_uuid: "6a90537cc4ed423b975af8b9cd87f711" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a90537cc4ed423b975af8b9cd87f711" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 37461 } health_report { overall_health: HEALTHY } } }
I20260925 10:40:04.359643 32147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:04.354288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38198 (local address 127.23.100.3:37461)
0925 10:40:04.354865 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:04.354871 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:04.354890 (+ 19us) server_negotiation.cc:407] Connection header received
0925 10:40:04.354971 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:04.354974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:04.355061 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:04.355225 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:40:04.355611 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:04.356415 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:04.357609 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:04.357994 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:04.358083 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:04.358510 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:40:04.359065 (+ 555us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:40:04.359169 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:04.359236 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 10:40:04.359289 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":269,"threads_started":1}
I20260925 10:40:04.372900 32135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:04.368332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:37103 (local address 127.23.100.2:35365)
0925 10:40:04.368474 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:04.368478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:04.368958 (+ 480us) server_negotiation.cc:407] Connection header received
0925 10:40:04.369082 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:04.369085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:04.369150 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:04.369217 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:40:04.369576 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:04.370219 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:04.371494 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:04.371875 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:04.371962 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:04.372328 (+ 366us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:40:04.372462 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:40:04.372528 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:04.372653 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 10:40:04.372712 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 10:40:04.373026 32150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:04.368166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.2:35365 (local address 127.23.100.3:37103)
0925 10:40:04.368882 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:04.368896 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:04.368993 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:04.369242 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:04.369244 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:04.369253 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:40:04.369465 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:04.369471 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:04.370360 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:04.370363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:04.371378 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:04.371389 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:04.372012 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:04.372016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:04.372140 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:04.372198 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:40:04.372548 (+ 350us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:40:04.372556 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 10:40:04.372618 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 10:40:04.372744 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":399,"threads_started":1}
I20260925 10:40:04.380420 31702 raft_consensus.cc:1261] T e4304ab12eaa4114908d192ef6fbb573 P dcafa1392bf049e4b508fb1ef2a74889 [term 1 FOLLOWER]: Refusing update from remote peer 6a90537cc4ed423b975af8b9cd87f711: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:40:04.380441 31840 raft_consensus.cc:1261] T e4304ab12eaa4114908d192ef6fbb573 P d78e073456484d9082e78231eee41e48 [term 1 FOLLOWER]: Refusing update from remote peer 6a90537cc4ed423b975af8b9cd87f711: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:40:04.380999 32061 consensus_queue.cc:1035] T e4304ab12eaa4114908d192ef6fbb573 P 6a90537cc4ed423b975af8b9cd87f711 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d78e073456484d9082e78231eee41e48" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 35365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:40:04.381256 32145 consensus_queue.cc:1035] T e4304ab12eaa4114908d192ef6fbb573 P 6a90537cc4ed423b975af8b9cd87f711 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcafa1392bf049e4b508fb1ef2a74889" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:40:04.407049 32147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:04.401742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:51113 (local address 127.23.100.3:37461)
0925 10:40:04.401882 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:04.401886 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:04.402458 (+ 572us) server_negotiation.cc:407] Connection header received
0925 10:40:04.402589 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:04.402594 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:04.402664 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:04.402735 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:40:04.403152 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:04.403950 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:04.405452 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:04.405846 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:04.405977 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:04.406366 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:40:04.406491 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:40:04.406585 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:04.406738 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 10:40:04.406795 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 10:40:04.407215 32156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:04.401704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.3:37461 (local address 127.23.100.2:51113)
0925 10:40:04.402355 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:04.402373 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:04.402488 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:04.402774 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:04.402778 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:04.402790 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:40:04.403007 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:04.403016 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:04.404074 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:04.404082 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:04.405276 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:04.405287 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:04.405984 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:04.405990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:04.406138 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:04.406172 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:40:04.406597 (+ 425us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:40:04.406608 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 10:40:04.406714 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 10:40:04.406844 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":354,"threads_started":1}
I20260925 10:40:04.436367 31545 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:40:04.436609 31545 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:40:04.439347 31545 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2620a3124bff4655ac88451061702e7d: Sending DeleteTablet for 3 replicas of tablet e4304ab12eaa4114908d192ef6fbb573
I20260925 10:40:04.440285 31820 tablet_service.cc:1514] Processing DeleteTablet for tablet e4304ab12eaa4114908d192ef6fbb573 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:40:04 UTC) from {username='oryx', principal='oryx/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:33060
I20260925 10:40:04.440369 31681 tablet_service.cc:1514] Processing DeleteTablet for tablet e4304ab12eaa4114908d192ef6fbb573 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:40:04 UTC) from {username='oryx', principal='oryx/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:51634
I20260925 10:40:04.440387 31959 tablet_service.cc:1514] Processing DeleteTablet for tablet e4304ab12eaa4114908d192ef6fbb573 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:40:04 UTC) from {username='oryx', principal='oryx/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:38188
I20260925 10:40:04.440672 32115 tablet_replica.cc:330] T e4304ab12eaa4114908d192ef6fbb573 P dcafa1392bf049e4b508fb1ef2a74889: stopping tablet replica
I20260925 10:40:04.440753 32116 tablet_replica.cc:330] T e4304ab12eaa4114908d192ef6fbb573 P d78e073456484d9082e78231eee41e48: stopping tablet replica
I20260925 10:40:04.440869 32115 raft_consensus.cc:2229] T e4304ab12eaa4114908d192ef6fbb573 P dcafa1392bf049e4b508fb1ef2a74889 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:40:04.440933 32116 raft_consensus.cc:2229] T e4304ab12eaa4114908d192ef6fbb573 P d78e073456484d9082e78231eee41e48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:40:04.441083 32115 raft_consensus.cc:2258] T e4304ab12eaa4114908d192ef6fbb573 P dcafa1392bf049e4b508fb1ef2a74889 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:40:04.441174 32116 raft_consensus.cc:2258] T e4304ab12eaa4114908d192ef6fbb573 P d78e073456484d9082e78231eee41e48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:40:04.441387 32117 tablet_replica.cc:330] T e4304ab12eaa4114908d192ef6fbb573 P 6a90537cc4ed423b975af8b9cd87f711: stopping tablet replica
I20260925 10:40:04.441572 32117 raft_consensus.cc:2229] T e4304ab12eaa4114908d192ef6fbb573 P 6a90537cc4ed423b975af8b9cd87f711 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:40:04.442041 32117 raft_consensus.cc:2258] T e4304ab12eaa4114908d192ef6fbb573 P 6a90537cc4ed423b975af8b9cd87f711 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:40:04.443077 32117 ts_tablet_manager.cc:1905] T e4304ab12eaa4114908d192ef6fbb573 P 6a90537cc4ed423b975af8b9cd87f711: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:40:04.443207 32116 ts_tablet_manager.cc:1905] T e4304ab12eaa4114908d192ef6fbb573 P d78e073456484d9082e78231eee41e48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:40:04.443768 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 31616
I20260925 10:40:04.450287 32117 ts_tablet_manager.cc:1918] T e4304ab12eaa4114908d192ef6fbb573 P 6a90537cc4ed423b975af8b9cd87f711: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:40:04.450836 32116 ts_tablet_manager.cc:1918] T e4304ab12eaa4114908d192ef6fbb573 P d78e073456484d9082e78231eee41e48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:40:04.450999 32116 log.cc:1196] T e4304ab12eaa4114908d192ef6fbb573 P d78e073456484d9082e78231eee41e48: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-1/wal/wals/e4304ab12eaa4114908d192ef6fbb573
I20260925 10:40:04.451157 32115 ts_tablet_manager.cc:1905] T e4304ab12eaa4114908d192ef6fbb573 P dcafa1392bf049e4b508fb1ef2a74889: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:40:04.451519 32116 ts_tablet_manager.cc:1939] T e4304ab12eaa4114908d192ef6fbb573 P d78e073456484d9082e78231eee41e48: Deleting consensus metadata
I20260925 10:40:04.452787 31531 catalog_manager.cc:4841] TS d78e073456484d9082e78231eee41e48 (127.23.100.2:35365): tablet e4304ab12eaa4114908d192ef6fbb573 (table test-table [id=758953fa9b234fad8d2e4cba93141c9e]) successfully deleted
I20260925 10:40:04.453722 32117 log.cc:1196] T e4304ab12eaa4114908d192ef6fbb573 P 6a90537cc4ed423b975af8b9cd87f711: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-2/wal/wals/e4304ab12eaa4114908d192ef6fbb573
I20260925 10:40:04.454238 32117 ts_tablet_manager.cc:1939] T e4304ab12eaa4114908d192ef6fbb573 P 6a90537cc4ed423b975af8b9cd87f711: Deleting consensus metadata
I20260925 10:40:04.455390 31531 catalog_manager.cc:4841] TS 6a90537cc4ed423b975af8b9cd87f711 (127.23.100.3:37461): tablet e4304ab12eaa4114908d192ef6fbb573 (table test-table [id=758953fa9b234fad8d2e4cba93141c9e]) successfully deleted
I20260925 10:40:04.458074 32115 ts_tablet_manager.cc:1918] T e4304ab12eaa4114908d192ef6fbb573 P dcafa1392bf049e4b508fb1ef2a74889: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:40:04.458213 32115 log.cc:1196] T e4304ab12eaa4114908d192ef6fbb573 P dcafa1392bf049e4b508fb1ef2a74889: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790332740062128-23952-0/minicluster-data/ts-0/wal/wals/e4304ab12eaa4114908d192ef6fbb573
I20260925 10:40:04.458676 32115 ts_tablet_manager.cc:1939] T e4304ab12eaa4114908d192ef6fbb573 P dcafa1392bf049e4b508fb1ef2a74889: Deleting consensus metadata
I20260925 10:40:04.459692 31529 catalog_manager.cc:4841] TS dcafa1392bf049e4b508fb1ef2a74889 (127.23.100.1:37725): tablet e4304ab12eaa4114908d192ef6fbb573 (table test-table [id=758953fa9b234fad8d2e4cba93141c9e]) successfully deleted
Sep 25 10:40:04 dist-test-slave-dxdd krb5kdc[31500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.100.1@KRBTEST.COM
I20260925 10:40:04.467947 32143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:04.445081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51656 (local address 127.23.100.1:37725)
0925 10:40:04.445252 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:04.445259 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:04.453530 (+ 8271us) server_negotiation.cc:407] Connection header received
0925 10:40:04.453844 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:04.453853 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:04.453953 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:04.454030 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:04.455834 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:04.456671 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:04.457833 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:04.458420 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:04.458538 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:04.462679 (+ 4141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:40:04.462694 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:40:04.462708 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:40:04.462736 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:40:04.465393 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:40:04.466006 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:40:04.466019 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:40:04.466021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:40:04.466156 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:40:04.466576 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:40:04.466582 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:40:04.466590 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:40:04.466897 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:40:04.467196 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:04.467453 (+ 257us) server_negotiation.cc:299] Negotiation successful
0925 10:40:04.467670 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 10:40:04.468497 31747 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:40:04.671100 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 31616
I20260925 10:40:04.702607 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 31756
Sep 25 10:40:04 dist-test-slave-dxdd krb5kdc[31500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.100.2@KRBTEST.COM
I20260925 10:40:04.722841 32135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:04.703399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33086 (local address 127.23.100.2:35365)
0925 10:40:04.703576 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:04.703582 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:04.703656 (+ 74us) server_negotiation.cc:407] Connection header received
0925 10:40:04.703857 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:04.703865 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:04.703960 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:04.704077 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:04.705575 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:04.706685 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:04.708492 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:04.709125 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:04.709265 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:04.714769 (+ 5504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:40:04.714790 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:40:04.714808 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:40:04.714861 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:40:04.718579 (+ 3718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:40:04.719439 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:40:04.719448 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:40:04.719452 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:40:04.719592 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:40:04.720059 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:40:04.720067 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:40:04.720071 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:40:04.720519 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:40:04.720817 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:04.722104 (+ 1287us) server_negotiation.cc:299] Negotiation successful
0925 10:40:04.722405 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 10:40:04.723670 31886 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:40:04.871500 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 31756
I20260925 10:40:04.896699 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 31895
Sep 25 10:40:04 dist-test-slave-dxdd krb5kdc[31500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.100.3@KRBTEST.COM
I20260925 10:40:04.911993 32147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:04.897572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38212 (local address 127.23.100.3:37461)
0925 10:40:04.897781 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:04.897787 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:04.897808 (+ 21us) server_negotiation.cc:407] Connection header received
0925 10:40:04.897876 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:04.897882 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:04.897952 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:04.898084 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:04.899403 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:04.900164 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:04.901402 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:04.901836 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:04.901975 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:04.906379 (+ 4404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:40:04.906390 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:40:04.906404 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:40:04.906432 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:40:04.909185 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:40:04.909792 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:40:04.909805 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:40:04.909808 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:40:04.909928 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:40:04.910283 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:40:04.910287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:40:04.910292 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:40:04.910802 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:40:04.911144 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:04.911354 (+ 210us) server_negotiation.cc:299] Negotiation successful
0925 10:40:04.911616 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 10:40:04.912818 32025 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:40:05.038800 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 31895
I20260925 10:40:05.064536 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 31513
I20260925 10:40:05.065671 31585 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:40:05.185518 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 31513
2026-09-25T10:40:05Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6805 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790332740062128-23952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:40:05 dist-test-slave-dxdd krb5kdc[32166](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:40:05 dist-test-slave-dxdd krb5kdc[32166](info): set up 2 sockets
Sep 25 10:40:05 dist-test-slave-dxdd krb5kdc[32166](info): commencing operation
krb5kdc: starting...
W20260925 10:40:06.271945 23952 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.003s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Sep 25 10:40:06 dist-test-slave-dxdd krb5kdc[32166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:40:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:40:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
I20260925 10:40:06.483435 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:36373
--webserver_interface=127.23.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:38123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.62:36373
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:40:06.793246 32179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:40:06.793504 32179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:40:06.793588 32179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:40:06.800643 32179 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:40:06.800736 32179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:40:06.800781 32179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:40:06.800840 32179 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:40:06.800889 32179 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:40:06.810143 32179 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:38123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790332740062128-23952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.62:36373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:36373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:40:06.811305 32179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:40:06.812481 32179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:40:06.824069 32185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:40:06.824035 32184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:40:06.824338 32187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:40:06.824733 32179 server_base.cc:1034] running on GCE node
I20260925 10:40:06.825505 32179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:40:06.826908 32179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:40:06.828109 32179 hybrid_clock.cc:648] HybridClock initialized: now 1790332806828062 us; error 41 us; skew 500 ppm
I20260925 10:40:06.828419 32179 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T10:40:06Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1714 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:40:06 dist-test-slave-dxdd krb5kdc[32193](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:40:06 dist-test-slave-dxdd krb5kdc[32193](info): set up 2 sockets
Sep 25 10:40:06 dist-test-slave-dxdd krb5kdc[32193](info): commencing operation
krb5kdc: starting...
W20260925 10:40:07.995524 23952 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.002s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Sep 25 10:40:08 dist-test-slave-dxdd krb5kdc[32193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:40:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:40:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
I20260925 10:40:08.213209 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:44673
--webserver_interface=127.23.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:41457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.62:44673
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:40:08.529536 32206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:40:08.529863 32206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:40:08.530047 32206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:40:08.537811 32206 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:40:08.537943 32206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:40:08.538041 32206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:40:08.538146 32206 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:40:08.538225 32206 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:40:08.547879 32206 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:41457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.62:44673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:44673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:40:08.549124 32206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:40:08.550446 32206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:40:08.561794 32211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:40:08.561842 32214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:40:08.561874 32212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:40:08.562636 32206 server_base.cc:1034] running on GCE node
I20260925 10:40:08.563345 32206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:40:08.564623 32206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:40:08.565852 32206 hybrid_clock.cc:648] HybridClock initialized: now 1790332808565822 us; error 44 us; skew 500 ppm
I20260925 10:40:08.566181 32206 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:40:08 dist-test-slave-dxdd krb5kdc[32193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332808, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:40:08.573629 32206 init.cc:377] Logged in from keytab as kudu/127.23.100.62@KRBTEST.COM (short username kudu)
I20260925 10:40:08.575063 32206 webserver.cc:460] Webserver started at http://127.23.100.62:33581/ using document root <none> and password file <none>
I20260925 10:40:08.575728 32206 fs_manager.cc:359] Metadata directory not provided
I20260925 10:40:08.575886 32206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:40:08.576179 32206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:40:08.579052 32206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/master-0/data/instance:
uuid: "8bf78f03bce34bbfb42574feca6722ef"
format_stamp: "Formatted at 2026-09-25 10:40:08 on dist-test-slave-dxdd"
server_key: "73b09fad2d6f896274d6f78d7226e49e"
server_key_iv: "1bb36c871fefca8c57ae3941d4dc2b24"
server_key_version: "encryptionkey@0"
I20260925 10:40:08.579787 32206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/master-0/wal/instance:
uuid: "8bf78f03bce34bbfb42574feca6722ef"
format_stamp: "Formatted at 2026-09-25 10:40:08 on dist-test-slave-dxdd"
server_key: "73b09fad2d6f896274d6f78d7226e49e"
server_key_iv: "1bb36c871fefca8c57ae3941d4dc2b24"
server_key_version: "encryptionkey@0"
I20260925 10:40:08.584793 32206 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260925 10:40:08.588099 32221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:40:08.589332 32206 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 10:40:08.589509 32206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/master-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/master-0/wal
uuid: "8bf78f03bce34bbfb42574feca6722ef"
format_stamp: "Formatted at 2026-09-25 10:40:08 on dist-test-slave-dxdd"
server_key: "73b09fad2d6f896274d6f78d7226e49e"
server_key_iv: "1bb36c871fefca8c57ae3941d4dc2b24"
server_key_version: "encryptionkey@0"
I20260925 10:40:08.589897 32206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:40:08.612551 32206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:40:08.616459 32206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:40:08.616758 32206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:40:08.634796 32206 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.62:44673
I20260925 10:40:08.634876 32272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.62:44673 every 8 connection(s)
I20260925 10:40:08.636149 32206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
I20260925 10:40:08.639663 32273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:40:08.643882 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 32206
I20260925 10:40:08.644112 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/master-0/wal/instance
I20260925 10:40:08.644739 23952 external_mini_cluster.cc:1428] Setting key 599ab5870745a3485efcdda7580cceb4
I20260925 10:40:08.649863 32273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bf78f03bce34bbfb42574feca6722ef: Bootstrap starting.
I20260925 10:40:08.653427 32273 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8bf78f03bce34bbfb42574feca6722ef: Neither blocks nor log segments found. Creating new log.
I20260925 10:40:08.654681 32273 log.cc:824] T 00000000000000000000000000000000 P 8bf78f03bce34bbfb42574feca6722ef: Log is configured to *not* fsync() on all Append() calls
I20260925 10:40:08.657950 32273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bf78f03bce34bbfb42574feca6722ef: No bootstrap required, opened a new log
Sep 25 10:40:08 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:40:08.665660 32273 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8bf78f03bce34bbfb42574feca6722ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bf78f03bce34bbfb42574feca6722ef" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 44673 } }
I20260925 10:40:08.666011 32273 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8bf78f03bce34bbfb42574feca6722ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:40:08.666123 32273 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8bf78f03bce34bbfb42574feca6722ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bf78f03bce34bbfb42574feca6722ef, State: Initialized, Role: FOLLOWER
I20260925 10:40:08.666782 32273 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bf78f03bce34bbfb42574feca6722ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bf78f03bce34bbfb42574feca6722ef" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 44673 } }
I20260925 10:40:08.667052 32273 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8bf78f03bce34bbfb42574feca6722ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:40:08.667184 32273 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8bf78f03bce34bbfb42574feca6722ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:40:08.667346 32273 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8bf78f03bce34bbfb42574feca6722ef [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:40:08.668172 32276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:08.646665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34712 (local address 127.23.100.62:44673)
0925 10:40:08.647484 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:08.647495 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:08.647537 (+ 42us) server_negotiation.cc:407] Connection header received
0925 10:40:08.649187 (+ 1650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:08.649207 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:08.649565 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:08.649867 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:08.651683 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:08.652939 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:08.654859 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:08.655980 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:08.656147 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:08.660713 (+ 4566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:40:08.660721 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:40:08.660733 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:40:08.660768 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:40:08.663926 (+ 3158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:40:08.664658 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:40:08.664664 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:40:08.664670 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:40:08.664835 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:40:08.665232 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:40:08.665237 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:40:08.665251 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:40:08.665719 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:40:08.666055 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:08.666467 (+ 412us) server_negotiation.cc:299] Negotiation successful
0925 10:40:08.666872 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":384,"threads_started":1}
I20260925 10:40:08.669780 32273 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8bf78f03bce34bbfb42574feca6722ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bf78f03bce34bbfb42574feca6722ef" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 44673 } }
I20260925 10:40:08.670228 32273 leader_election.cc:302] T 00000000000000000000000000000000 P 8bf78f03bce34bbfb42574feca6722ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8bf78f03bce34bbfb42574feca6722ef; no voters:
I20260925 10:40:08.670945 32273 leader_election.cc:288] T 00000000000000000000000000000000 P 8bf78f03bce34bbfb42574feca6722ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:40:08.671043 32278 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8bf78f03bce34bbfb42574feca6722ef [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:40:08.671790 32278 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8bf78f03bce34bbfb42574feca6722ef [term 1 LEADER]: Becoming Leader. State: Replica: 8bf78f03bce34bbfb42574feca6722ef, State: Running, Role: LEADER
I20260925 10:40:08.672304 32278 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bf78f03bce34bbfb42574feca6722ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bf78f03bce34bbfb42574feca6722ef" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 44673 } }
I20260925 10:40:08.673013 32273 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8bf78f03bce34bbfb42574feca6722ef [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:40:08.674922 32280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bf78f03bce34bbfb42574feca6722ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bf78f03bce34bbfb42574feca6722ef. Latest consensus state: current_term: 1 leader_uuid: "8bf78f03bce34bbfb42574feca6722ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bf78f03bce34bbfb42574feca6722ef" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 44673 } } }
I20260925 10:40:08.675235 32280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bf78f03bce34bbfb42574feca6722ef [sys.catalog]: This master's current role is: LEADER
I20260925 10:40:08.675941 32279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bf78f03bce34bbfb42574feca6722ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8bf78f03bce34bbfb42574feca6722ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bf78f03bce34bbfb42574feca6722ef" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 44673 } } }
I20260925 10:40:08.676234 32279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bf78f03bce34bbfb42574feca6722ef [sys.catalog]: This master's current role is: LEADER
I20260925 10:40:08.677093 32285 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:40:08.683300 32285 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:40:08.693259 32285 catalog_manager.cc:1347] Generated new cluster ID: 57106abb06684edbb14bfb436f3fe9be
I20260925 10:40:08.693367 32285 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:40:08.709071 32285 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:40:08.710861 32285 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:40:08.720278 32285 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8bf78f03bce34bbfb42574feca6722ef: Generated new TSK 0
I20260925 10:40:08.721228 32285 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab.
Entry for principal kudu/127.23.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab.
I20260925 10:40:08.771915 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.1:0
--local_ip_for_outbound_sockets=127.23.100.1
--webserver_interface=127.23.100.1
--webserver_port=0
--tserver_master_addrs=127.23.100.62:44673
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab
--principal=kudu/127.23.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:41457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:40:09.089562 32299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:40:09.089807 32299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:40:09.089983 32299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:40:09.097249 32299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:40:09.097352 32299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:40:09.097548 32299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.1
I20260925 10:40:09.106999 32299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:41457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab
--principal=kudu/127.23.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.1
--webserver_port=0
--tserver_master_addrs=127.23.100.62:44673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.1
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:40:09.108232 32299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:40:09.109462 32299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:40:09.122113 32305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:40:09.122241 32304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:40:09.122128 32307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:40:09.122881 32299 server_base.cc:1034] running on GCE node
I20260925 10:40:09.123423 32299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:40:09.124212 32299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:40:09.125408 32299 hybrid_clock.cc:648] HybridClock initialized: now 1790332809125373 us; error 25 us; skew 500 ppm
I20260925 10:40:09.125715 32299 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:40:09 dist-test-slave-dxdd krb5kdc[32193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332809, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:40:09.128614 32299 init.cc:377] Logged in from keytab as kudu/127.23.100.1@KRBTEST.COM (short username kudu)
I20260925 10:40:09.129995 32299 webserver.cc:460] Webserver started at http://127.23.100.1:33403/ using document root <none> and password file <none>
I20260925 10:40:09.130695 32299 fs_manager.cc:359] Metadata directory not provided
I20260925 10:40:09.130874 32299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:40:09.131187 32299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:40:09.133894 32299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-0/data/instance:
uuid: "eae40650228d40d5bf2d84480f88f0cb"
format_stamp: "Formatted at 2026-09-25 10:40:09 on dist-test-slave-dxdd"
server_key: "8b0110a2e51c22b3bdf8e8dd5ffe8fed"
server_key_iv: "c12544f0341304d019ce32267c33ba9c"
server_key_version: "encryptionkey@0"
I20260925 10:40:09.134675 32299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-0/wal/instance:
uuid: "eae40650228d40d5bf2d84480f88f0cb"
format_stamp: "Formatted at 2026-09-25 10:40:09 on dist-test-slave-dxdd"
server_key: "8b0110a2e51c22b3bdf8e8dd5ffe8fed"
server_key_iv: "c12544f0341304d019ce32267c33ba9c"
server_key_version: "encryptionkey@0"
I20260925 10:40:09.139606 32299 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 10:40:09.142822 32314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:40:09.144006 32299 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.002s
I20260925 10:40:09.144174 32299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-0/wal
uuid: "eae40650228d40d5bf2d84480f88f0cb"
format_stamp: "Formatted at 2026-09-25 10:40:09 on dist-test-slave-dxdd"
server_key: "8b0110a2e51c22b3bdf8e8dd5ffe8fed"
server_key_iv: "c12544f0341304d019ce32267c33ba9c"
server_key_version: "encryptionkey@0"
I20260925 10:40:09.144361 32299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:40:09.154640 32299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:40:09.158437 32299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:40:09.158715 32299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:40:09.159511 32299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:40:09.160760 32299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:40:09.160831 32299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:40:09.160988 32299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:40:09.161046 32299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:40:09.194622 32299 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.1:45675
I20260925 10:40:09.194701 32426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.1:45675 every 8 connection(s)
I20260925 10:40:09.195859 32299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
I20260925 10:40:09.200999 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 32299
I20260925 10:40:09.201216 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-0/wal/instance
I20260925 10:40:09.201822 23952 external_mini_cluster.cc:1428] Setting key a12b3a88cf36089997d2c2f775d4a5c7
Sep 25 10:40:09 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332809, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.1@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:40:09.215770 32429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:09.198275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:35143 (local address 127.23.100.62:44673)
0925 10:40:09.198748 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:09.198752 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:09.199642 (+ 890us) server_negotiation.cc:407] Connection header received
0925 10:40:09.201444 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:09.201450 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:09.201535 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:09.201654 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:09.203277 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:09.204064 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:09.205603 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:09.206005 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:09.206161 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:09.209452 (+ 3291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:40:09.209457 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:40:09.209460 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:40:09.209511 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:40:09.212644 (+ 3133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:40:09.213515 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:40:09.213519 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:40:09.213522 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:40:09.213642 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:40:09.214089 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:40:09.214092 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:40:09.214094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:40:09.214457 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:40:09.214566 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:09.215371 (+ 805us) server_negotiation.cc:299] Negotiation successful
0925 10:40:09.215523 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":233,"threads_started":1}
I20260925 10:40:09.217262 32430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:09.198828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44673 (local address 127.23.100.1:35143)
0925 10:40:09.199481 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:09.199517 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:09.201163 (+ 1646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:09.201872 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:09.201879 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:09.202476 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:09.203117 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:09.203133 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:09.204215 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:09.204220 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:09.205423 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:09.205438 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:09.206240 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:09.206249 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:09.206519 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:09.207237 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:09.207258 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:09.209238 (+ 1980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:40:09.212773 (+ 3535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:40:09.212784 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:40:09.212801 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:40:09.213401 (+ 600us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:40:09.213741 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:40:09.213748 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:40:09.213754 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:40:09.213980 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:40:09.214589 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:40:09.214599 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:40:09.215014 (+ 415us) client_negotiation.cc:769] Sending connection context
0925 10:40:09.215295 (+ 281us) client_negotiation.cc:240] Negotiation successful
0925 10:40:09.215872 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":258,"threads_started":1}
I20260925 10:40:09.218804 32427 heartbeater.cc:347] Connected to a master server at 127.23.100.62:44673
I20260925 10:40:09.219098 32427 heartbeater.cc:464] Registering TS with master...
I20260925 10:40:09.219846 32427 heartbeater.cc:511] Master 127.23.100.62:44673 requested a full tablet report, sending...
I20260925 10:40:09.221947 32238 ts_manager.cc:194] Registered new tserver with Master: eae40650228d40d5bf2d84480f88f0cb (127.23.100.1:45675)
I20260925 10:40:09.223410 32238 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.1@KRBTEST.COM'} at 127.23.100.1:35143
WARNING: no policy specified for kudu/127.23.100.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab.
Entry for principal kudu/127.23.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab.
I20260925 10:40:09.251353 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.2:0
--local_ip_for_outbound_sockets=127.23.100.2
--webserver_interface=127.23.100.2
--webserver_port=0
--tserver_master_addrs=127.23.100.62:44673
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab
--principal=kudu/127.23.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:41457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:40:09.560555 32433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:40:09.560801 32433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:40:09.560887 32433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:40:09.567531 32433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:40:09.567636 32433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:40:09.567747 32433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.2
I20260925 10:40:09.577040 32433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:41457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab
--principal=kudu/127.23.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.2
--webserver_port=0
--tserver_master_addrs=127.23.100.62:44673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.2
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:40:09.578162 32433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:40:09.579344 32433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:40:09.591924 32441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:40:09.591952 32439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:40:09.592208 32438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:40:09.592706 32433 server_base.cc:1034] running on GCE node
I20260925 10:40:09.593420 32433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:40:09.594383 32433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:40:09.595610 32433 hybrid_clock.cc:648] HybridClock initialized: now 1790332809595538 us; error 56 us; skew 500 ppm
I20260925 10:40:09.595929 32433 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:40:09 dist-test-slave-dxdd krb5kdc[32193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332809, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:40:09.599057 32433 init.cc:377] Logged in from keytab as kudu/127.23.100.2@KRBTEST.COM (short username kudu)
I20260925 10:40:09.600437 32433 webserver.cc:460] Webserver started at http://127.23.100.2:33525/ using document root <none> and password file <none>
I20260925 10:40:09.601100 32433 fs_manager.cc:359] Metadata directory not provided
I20260925 10:40:09.601240 32433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:40:09.601528 32433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:40:09.604354 32433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-1/data/instance:
uuid: "0ba17c512821475c943ab3f206c027b5"
format_stamp: "Formatted at 2026-09-25 10:40:09 on dist-test-slave-dxdd"
server_key: "0f1980c9076b839ebc4c973d5b6b9dab"
server_key_iv: "86f73d0cae81ccd35519252b123ad1ff"
server_key_version: "encryptionkey@0"
I20260925 10:40:09.605113 32433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-1/wal/instance:
uuid: "0ba17c512821475c943ab3f206c027b5"
format_stamp: "Formatted at 2026-09-25 10:40:09 on dist-test-slave-dxdd"
server_key: "0f1980c9076b839ebc4c973d5b6b9dab"
server_key_iv: "86f73d0cae81ccd35519252b123ad1ff"
server_key_version: "encryptionkey@0"
I20260925 10:40:09.609886 32433 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 10:40:09.613142 32448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:40:09.614395 32433 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 10:40:09.614557 32433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-1/wal
uuid: "0ba17c512821475c943ab3f206c027b5"
format_stamp: "Formatted at 2026-09-25 10:40:09 on dist-test-slave-dxdd"
server_key: "0f1980c9076b839ebc4c973d5b6b9dab"
server_key_iv: "86f73d0cae81ccd35519252b123ad1ff"
server_key_version: "encryptionkey@0"
I20260925 10:40:09.614749 32433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:40:09.633402 32433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:40:09.637426 32433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:40:09.637698 32433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:40:09.638680 32433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:40:09.639914 32433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:40:09.640027 32433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:40:09.640182 32433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:40:09.640252 32433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:40:09.674052 32433 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.2:39099
I20260925 10:40:09.674175 32560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.2:39099 every 8 connection(s)
I20260925 10:40:09.675273 32433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
I20260925 10:40:09.679533 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 32433
I20260925 10:40:09.679762 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-1/wal/instance
I20260925 10:40:09.680214 23952 external_mini_cluster.cc:1428] Setting key 2533aae32d41a9b49666bd177141b781
Sep 25 10:40:09 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332809, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.2@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:40:09.695473 32429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:09.677662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:36157 (local address 127.23.100.62:44673)
0925 10:40:09.677871 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:09.677877 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:09.679100 (+ 1223us) server_negotiation.cc:407] Connection header received
0925 10:40:09.681160 (+ 2060us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:09.681166 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:09.681248 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:09.681350 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:09.682892 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:09.684100 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:09.685560 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:09.686310 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:09.686484 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:09.689775 (+ 3291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:40:09.689780 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:40:09.689783 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:40:09.689813 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:40:09.692373 (+ 2560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:40:09.693058 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:40:09.693065 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:40:09.693071 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:40:09.693243 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:40:09.693702 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:40:09.693709 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:40:09.693712 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:40:09.693971 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:40:09.694121 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:09.694912 (+ 791us) server_negotiation.cc:299] Negotiation successful
0925 10:40:09.695137 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 10:40:09.696313 32563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:09.678136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44673 (local address 127.23.100.2:36157)
0925 10:40:09.678900 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:09.678976 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:09.680832 (+ 1856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:09.681529 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:09.681536 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:09.682132 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:09.682705 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:09.682722 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:09.684263 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:09.684267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:09.685416 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:09.685427 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:09.686534 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:09.686539 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:09.686713 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:09.687355 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:09.687380 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:09.689571 (+ 2191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:40:09.692525 (+ 2954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:40:09.692535 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:40:09.692550 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:40:09.692940 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:40:09.693355 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:40:09.693360 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:40:09.693363 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:40:09.693600 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:40:09.694100 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:40:09.694106 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:40:09.694529 (+ 423us) client_negotiation.cc:769] Sending connection context
0925 10:40:09.694837 (+ 308us) client_negotiation.cc:240] Negotiation successful
0925 10:40:09.695304 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":365,"threads_started":1}
I20260925 10:40:09.697845 32561 heartbeater.cc:347] Connected to a master server at 127.23.100.62:44673
I20260925 10:40:09.698166 32561 heartbeater.cc:464] Registering TS with master...
I20260925 10:40:09.698874 32561 heartbeater.cc:511] Master 127.23.100.62:44673 requested a full tablet report, sending...
I20260925 10:40:09.700323 32238 ts_manager.cc:194] Registered new tserver with Master: 0ba17c512821475c943ab3f206c027b5 (127.23.100.2:39099)
I20260925 10:40:09.701123 32238 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.2@KRBTEST.COM'} at 127.23.100.2:36157
WARNING: no policy specified for kudu/127.23.100.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab.
Entry for principal kudu/127.23.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab.
I20260925 10:40:09.726684 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.3:0
--local_ip_for_outbound_sockets=127.23.100.3
--webserver_interface=127.23.100.3
--webserver_port=0
--tserver_master_addrs=127.23.100.62:44673
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab
--principal=kudu/127.23.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:41457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:40:10.037423 32566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:40:10.037716 32566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:40:10.037889 32566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:40:10.044693 32566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:40:10.044829 32566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:40:10.045017 32566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.3
I20260925 10:40:10.054342 32566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:41457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab
--principal=kudu/127.23.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.3
--webserver_port=0
--tserver_master_addrs=127.23.100.62:44673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.3
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:40:10.055505 32566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:40:10.056675 32566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:40:10.068830 32571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:40:10.068794 32572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:40:10.068861 32574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:40:10.069391 32566 server_base.cc:1034] running on GCE node
I20260925 10:40:10.069969 32566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:40:10.070883 32566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:40:10.072119 32566 hybrid_clock.cc:648] HybridClock initialized: now 1790332810072064 us; error 49 us; skew 500 ppm
I20260925 10:40:10.072413 32566 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:40:10 dist-test-slave-dxdd krb5kdc[32193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332810, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:40:10.075492 32566 init.cc:377] Logged in from keytab as kudu/127.23.100.3@KRBTEST.COM (short username kudu)
I20260925 10:40:10.076725 32566 webserver.cc:460] Webserver started at http://127.23.100.3:41447/ using document root <none> and password file <none>
I20260925 10:40:10.077327 32566 fs_manager.cc:359] Metadata directory not provided
I20260925 10:40:10.077466 32566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:40:10.077778 32566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:40:10.080618 32566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-2/data/instance:
uuid: "ca8b76b6ee7c4f6aa82b94660110d1b3"
format_stamp: "Formatted at 2026-09-25 10:40:10 on dist-test-slave-dxdd"
server_key: "054581121c1ee56691a6ed69b152f13b"
server_key_iv: "334039a83706e3eb71d223318020e503"
server_key_version: "encryptionkey@0"
I20260925 10:40:10.081351 32566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-2/wal/instance:
uuid: "ca8b76b6ee7c4f6aa82b94660110d1b3"
format_stamp: "Formatted at 2026-09-25 10:40:10 on dist-test-slave-dxdd"
server_key: "054581121c1ee56691a6ed69b152f13b"
server_key_iv: "334039a83706e3eb71d223318020e503"
server_key_version: "encryptionkey@0"
I20260925 10:40:10.086045 32566 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 10:40:10.089408 32581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:40:10.090728 32566 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:40:10.090898 32566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-2/wal
uuid: "ca8b76b6ee7c4f6aa82b94660110d1b3"
format_stamp: "Formatted at 2026-09-25 10:40:10 on dist-test-slave-dxdd"
server_key: "054581121c1ee56691a6ed69b152f13b"
server_key_iv: "334039a83706e3eb71d223318020e503"
server_key_version: "encryptionkey@0"
I20260925 10:40:10.091128 32566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:40:10.121856 32566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:40:10.126181 32566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:40:10.126499 32566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:40:10.127398 32566 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:40:10.128645 32566 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:40:10.128747 32566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:40:10.128888 32566 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:40:10.128968 32566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:40:10.162129 32566 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.3:45801
I20260925 10:40:10.162215 32693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.3:45801 every 8 connection(s)
I20260925 10:40:10.163513 32566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
I20260925 10:40:10.164295 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 32566
I20260925 10:40:10.164472 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-2/wal/instance
I20260925 10:40:10.165045 23952 external_mini_cluster.cc:1428] Setting key 2f6fab383634cf4cbb8cc7439b78db11
Sep 25 10:40:10 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332810, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.3@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:40:10.183348 32429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:10.166448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:58087 (local address 127.23.100.62:44673)
0925 10:40:10.166630 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:10.166634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:10.168111 (+ 1477us) server_negotiation.cc:407] Connection header received
0925 10:40:10.169889 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:10.169893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:10.169959 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:10.170041 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:10.171491 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:10.172199 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:10.173713 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:10.174133 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:10.174225 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:10.177501 (+ 3276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:40:10.177506 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:40:10.177509 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:40:10.177537 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:40:10.180842 (+ 3305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:40:10.181503 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:40:10.181507 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:40:10.181509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:40:10.181618 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:40:10.181965 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:40:10.181968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:40:10.181970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:40:10.182231 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:40:10.182391 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:10.182964 (+ 573us) server_negotiation.cc:299] Negotiation successful
0925 10:40:10.183117 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 10:40:10.184283 32696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:10.167134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44673 (local address 127.23.100.3:58087)
0925 10:40:10.167954 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:10.167997 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:10.169652 (+ 1655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:10.170241 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:10.170248 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:10.170813 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:10.171335 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:10.171351 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:10.172325 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:10.172328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:10.173598 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:10.173608 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:10.174363 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:10.174370 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:10.174599 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:10.175257 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:10.175278 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:10.177325 (+ 2047us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:40:10.180955 (+ 3630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:40:10.180961 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:40:10.180972 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:40:10.181399 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:40:10.181704 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:40:10.181707 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:40:10.181709 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:40:10.181873 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:40:10.182370 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:40:10.182375 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:40:10.182673 (+ 298us) client_negotiation.cc:769] Sending connection context
0925 10:40:10.182884 (+ 211us) client_negotiation.cc:240] Negotiation successful
0925 10:40:10.183215 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":360,"threads_started":1}
I20260925 10:40:10.185614 32694 heartbeater.cc:347] Connected to a master server at 127.23.100.62:44673
I20260925 10:40:10.185896 32694 heartbeater.cc:464] Registering TS with master...
I20260925 10:40:10.186625 32694 heartbeater.cc:511] Master 127.23.100.62:44673 requested a full tablet report, sending...
I20260925 10:40:10.188031 32238 ts_manager.cc:194] Registered new tserver with Master: ca8b76b6ee7c4f6aa82b94660110d1b3 (127.23.100.3:45801)
I20260925 10:40:10.188777 32238 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.3@KRBTEST.COM'} at 127.23.100.3:58087
I20260925 10:40:10.192683 23952 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.23.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.100.1@KRBTEST.COM" created.
I20260925 10:40:10.226258 32427 heartbeater.cc:503] Master 127.23.100.62:44673 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/oryx_127.23.100.1.keytab.
Entry for principal oryx/127.23.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/oryx_127.23.100.1.keytab.
WARNING: no policy specified for oryx/127.23.100.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.100.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/oryx_127.23.100.2.keytab.
Entry for principal oryx/127.23.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/oryx_127.23.100.2.keytab.
WARNING: no policy specified for oryx/127.23.100.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.100.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/oryx_127.23.100.3.keytab.
Entry for principal oryx/127.23.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/oryx_127.23.100.3.keytab.
I20260925 10:40:10.315111 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 32299
Sep 25 10:40:10 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.1@KRBTEST.COM
I20260925 10:40:10.330446 32704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:10.316259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52146 (local address 127.23.100.1:45675)
0925 10:40:10.316868 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:10.316876 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:10.316902 (+ 26us) server_negotiation.cc:407] Connection header received
0925 10:40:10.316962 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:10.316968 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:10.317175 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:10.317301 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:10.318760 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:10.319830 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:10.321015 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:10.321543 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:10.321644 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:10.325381 (+ 3737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:40:10.325391 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:40:10.325404 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:40:10.325438 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:40:10.328219 (+ 2781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:40:10.328762 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:40:10.328769 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:40:10.328771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:40:10.328888 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:40:10.329190 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:40:10.329193 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:40:10.329195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:40:10.329508 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:40:10.329735 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:10.329923 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 10:40:10.330097 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":287,"threads_started":1}
I20260925 10:40:10.331653 32422 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:40:10.413878 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 32299
I20260925 10:40:10.429924 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 32433
Sep 25 10:40:10 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.2@KRBTEST.COM
I20260925 10:40:10.445232 32706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:10.430807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44340 (local address 127.23.100.2:39099)
0925 10:40:10.431296 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:10.431302 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:10.431324 (+ 22us) server_negotiation.cc:407] Connection header received
0925 10:40:10.431380 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:10.431386 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:10.431585 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:10.431758 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:10.432960 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:10.433909 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:10.435224 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:10.435757 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:10.435888 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:10.439988 (+ 4100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:40:10.440001 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:40:10.440019 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:40:10.440055 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:40:10.442618 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:40:10.443238 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:40:10.443244 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:40:10.443246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:40:10.443356 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:40:10.443710 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:40:10.443716 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:40:10.443719 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:40:10.444030 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:40:10.444363 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:10.444656 (+ 293us) server_negotiation.cc:299] Negotiation successful
0925 10:40:10.444899 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":222,"threads_started":1}
I20260925 10:40:10.446471 32556 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:40:10.526962 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 32433
I20260925 10:40:10.542514 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 32566
Sep 25 10:40:10 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.3@KRBTEST.COM
I20260925 10:40:10.557719 32708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:10.543532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32852 (local address 127.23.100.3:45801)
0925 10:40:10.544086 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:10.544093 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:10.544120 (+ 27us) server_negotiation.cc:407] Connection header received
0925 10:40:10.544183 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:10.544190 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:10.544394 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:10.544518 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:10.545732 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:10.546785 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:10.547996 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:10.548531 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:10.548629 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:10.552829 (+ 4200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:40:10.552839 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:40:10.552853 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:40:10.552883 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:40:10.555624 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:40:10.556154 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:40:10.556160 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:40:10.556162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:40:10.556268 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:40:10.556557 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:40:10.556559 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:40:10.556561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:40:10.556857 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:40:10.557092 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:10.557263 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 10:40:10.557442 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":252,"threads_started":1}
I20260925 10:40:10.559151 32689 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:40:10.640591 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 32566
I20260925 10:40:10.656495 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 32206
I20260925 10:40:10.657809 32268 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:40:10.762498 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 32206
I20260925 10:40:10.782732 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:44673
--webserver_interface=127.23.100.62
--webserver_port=33581
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:41457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.62:44673
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:40:11.101053 32711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:40:11.101322 32711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:40:11.101406 32711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:40:11.108296 32711 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:40:11.108409 32711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:40:11.108454 32711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:40:11.108505 32711 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:40:11.108548 32711 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:40:11.117605 32711 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:41457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.62:44673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:44673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.62
--webserver_port=33581
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:40:11.118745 32711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:40:11.120018 32711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:40:11.131608 32719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:40:11.131649 32717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:40:11.131691 32716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:40:11.135667 32711 server_base.cc:1034] running on GCE node
I20260925 10:40:11.136382 32711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:40:11.137681 32711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:40:11.138897 32711 hybrid_clock.cc:648] HybridClock initialized: now 1790332811138868 us; error 56 us; skew 500 ppm
I20260925 10:40:11.139252 32711 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:40:11 dist-test-slave-dxdd krb5kdc[32193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332811, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:40:11.142256 32711 init.cc:377] Logged in from keytab as kudu/127.23.100.62@KRBTEST.COM (short username kudu)
I20260925 10:40:11.143762 32711 webserver.cc:460] Webserver started at http://127.23.100.62:33581/ using document root <none> and password file <none>
I20260925 10:40:11.144438 32711 fs_manager.cc:359] Metadata directory not provided
I20260925 10:40:11.144593 32711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:40:11.149595 32711 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:40:11.152333 32726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:40:11.153522 32711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:40:11.153707 32711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/master-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/master-0/wal
uuid: "8bf78f03bce34bbfb42574feca6722ef"
format_stamp: "Formatted at 2026-09-25 10:40:08 on dist-test-slave-dxdd"
server_key: "73b09fad2d6f896274d6f78d7226e49e"
server_key_iv: "1bb36c871fefca8c57ae3941d4dc2b24"
server_key_version: "encryptionkey@0"
I20260925 10:40:11.154719 32711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:40:11.170403 32711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:40:11.174507 32711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:40:11.174764 32711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:40:11.192559 32711 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.62:44673
I20260925 10:40:11.192598 309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.62:44673 every 8 connection(s)
I20260925 10:40:11.193969 32711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
I20260925 10:40:11.200290 310 sys_catalog.cc:263] Verifying existing consensus state
I20260925 10:40:11.202874 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 32711
I20260925 10:40:11.203861 310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bf78f03bce34bbfb42574feca6722ef: Bootstrap starting.
I20260925 10:40:11.203872 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.1:45675
--local_ip_for_outbound_sockets=127.23.100.1
--tserver_master_addrs=127.23.100.62:44673
--webserver_port=33403
--webserver_interface=127.23.100.1
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab
--principal=kudu/127.23.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:41457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.23.100.1
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/oryx_127.23.100.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:40:11.224225 310 log.cc:824] T 00000000000000000000000000000000 P 8bf78f03bce34bbfb42574feca6722ef: Log is configured to *not* fsync() on all Append() calls
I20260925 10:40:11.232748 310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bf78f03bce34bbfb42574feca6722ef: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:40:11.233247 310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bf78f03bce34bbfb42574feca6722ef: Bootstrap complete.
I20260925 10:40:11.242236 310 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8bf78f03bce34bbfb42574feca6722ef [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bf78f03bce34bbfb42574feca6722ef" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 44673 } }
I20260925 10:40:11.243101 310 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8bf78f03bce34bbfb42574feca6722ef [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bf78f03bce34bbfb42574feca6722ef, State: Initialized, Role: FOLLOWER
I20260925 10:40:11.243701 310 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bf78f03bce34bbfb42574feca6722ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bf78f03bce34bbfb42574feca6722ef" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 44673 } }
I20260925 10:40:11.243924 310 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8bf78f03bce34bbfb42574feca6722ef [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:40:11.244040 310 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8bf78f03bce34bbfb42574feca6722ef [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:40:11.244244 310 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8bf78f03bce34bbfb42574feca6722ef [term 1 FOLLOWER]: Advancing to term 2
I20260925 10:40:11.247838 310 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8bf78f03bce34bbfb42574feca6722ef [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bf78f03bce34bbfb42574feca6722ef" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 44673 } }
I20260925 10:40:11.248317 310 leader_election.cc:302] T 00000000000000000000000000000000 P 8bf78f03bce34bbfb42574feca6722ef [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8bf78f03bce34bbfb42574feca6722ef; no voters:
I20260925 10:40:11.248965 310 leader_election.cc:288] T 00000000000000000000000000000000 P 8bf78f03bce34bbfb42574feca6722ef [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 10:40:11.249099 314 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8bf78f03bce34bbfb42574feca6722ef [term 2 FOLLOWER]: Leader election won for term 2
I20260925 10:40:11.249938 314 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8bf78f03bce34bbfb42574feca6722ef [term 2 LEADER]: Becoming Leader. State: Replica: 8bf78f03bce34bbfb42574feca6722ef, State: Running, Role: LEADER
I20260925 10:40:11.250547 314 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bf78f03bce34bbfb42574feca6722ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bf78f03bce34bbfb42574feca6722ef" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 44673 } }
I20260925 10:40:11.251044 310 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8bf78f03bce34bbfb42574feca6722ef [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:40:11.253162 316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bf78f03bce34bbfb42574feca6722ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bf78f03bce34bbfb42574feca6722ef. Latest consensus state: current_term: 2 leader_uuid: "8bf78f03bce34bbfb42574feca6722ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bf78f03bce34bbfb42574feca6722ef" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 44673 } } }
I20260925 10:40:11.253561 316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bf78f03bce34bbfb42574feca6722ef [sys.catalog]: This master's current role is: LEADER
I20260925 10:40:11.255381 319 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:40:11.255190 315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bf78f03bce34bbfb42574feca6722ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8bf78f03bce34bbfb42574feca6722ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bf78f03bce34bbfb42574feca6722ef" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 44673 } } }
I20260925 10:40:11.255569 315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bf78f03bce34bbfb42574feca6722ef [sys.catalog]: This master's current role is: LEADER
I20260925 10:40:11.261439 319 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:40:11.262768 319 catalog_manager.cc:1259] Loaded cluster ID: 57106abb06684edbb14bfb436f3fe9be
I20260925 10:40:11.262904 319 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:40:11.266776 319 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:40:11.268231 319 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8bf78f03bce34bbfb42574feca6722ef: Loaded TSK: 0
I20260925 10:40:11.269680 319 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 10:40:11.532088 312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:40:11.532378 312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:40:11.532541 312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:40:11.539566 312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:40:11.539702 312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:40:11.539860 312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.1
I20260925 10:40:11.549203 312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:41457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/oryx_127.23.100.1.keytab
--principal=oryx/127.23.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.1:45675
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.1
--webserver_port=33403
--tserver_master_addrs=127.23.100.62:44673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.1
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:40:11.550472 312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:40:11.551755 312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:40:11.565691 337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:40:11.566390 336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:40:11.566570 339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:40:11.566810 312 server_base.cc:1034] running on GCE node
I20260925 10:40:11.567556 312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:40:11.568553 312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:40:11.569765 312 hybrid_clock.cc:648] HybridClock initialized: now 1790332811569711 us; error 46 us; skew 500 ppm
I20260925 10:40:11.570071 312 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:40:11 dist-test-slave-dxdd krb5kdc[32193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332811, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:40:11.573109 312 init.cc:377] Logged in from keytab as oryx/127.23.100.1@KRBTEST.COM (short username oryx)
I20260925 10:40:11.574651 312 webserver.cc:460] Webserver started at http://127.23.100.1:33403/ using document root <none> and password file <none>
I20260925 10:40:11.575325 312 fs_manager.cc:359] Metadata directory not provided
I20260925 10:40:11.575465 312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:40:11.580649 312 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260925 10:40:11.583343 346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:40:11.584726 312 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 10:40:11.584888 312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-0/wal
uuid: "eae40650228d40d5bf2d84480f88f0cb"
format_stamp: "Formatted at 2026-09-25 10:40:09 on dist-test-slave-dxdd"
server_key: "8b0110a2e51c22b3bdf8e8dd5ffe8fed"
server_key_iv: "c12544f0341304d019ce32267c33ba9c"
server_key_version: "encryptionkey@0"
I20260925 10:40:11.585881 312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:40:11.605115 312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:40:11.609527 312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:40:11.609745 312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:40:11.610731 312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:40:11.611996 312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:40:11.612098 312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:40:11.612172 312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:40:11.612232 312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:40:11.645466 312 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.1:45675
I20260925 10:40:11.645511 458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.1:45675 every 8 connection(s)
I20260925 10:40:11.646908 312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
I20260925 10:40:11.657260 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 312
I20260925 10:40:11.658007 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.2:39099
--local_ip_for_outbound_sockets=127.23.100.2
--tserver_master_addrs=127.23.100.62:44673
--webserver_port=33525
--webserver_interface=127.23.100.2
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab
--principal=kudu/127.23.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:41457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.23.100.2
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/oryx_127.23.100.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 10:40:11 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.1@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:40:11 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.1@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 10:40:11.997552 463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:40:11.997852 463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:40:11.998011 463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:40:12.005100 463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:40:12.005249 463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:40:12.005414 463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.2
I20260925 10:40:12.014858 463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:41457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/oryx_127.23.100.2.keytab
--principal=oryx/127.23.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.2:39099
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.2
--webserver_port=33525
--tserver_master_addrs=127.23.100.62:44673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.2
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:40:12.016098 463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:40:12.017349 463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:40:12.038676 470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:40:12.038733 469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:40:12.040138 472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:40:12.040863 463 server_base.cc:1034] running on GCE node
I20260925 10:40:12.041483 463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:40:12.042430 463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:40:12.043715 463 hybrid_clock.cc:648] HybridClock initialized: now 1790332812043640 us; error 76 us; skew 500 ppm
I20260925 10:40:12.044021 463 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:40:12 dist-test-slave-dxdd krb5kdc[32193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332812, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:40:12.047533 463 init.cc:377] Logged in from keytab as oryx/127.23.100.2@KRBTEST.COM (short username oryx)
I20260925 10:40:12.049156 463 webserver.cc:460] Webserver started at http://127.23.100.2:33525/ using document root <none> and password file <none>
I20260925 10:40:12.049809 463 fs_manager.cc:359] Metadata directory not provided
I20260925 10:40:12.049952 463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:40:12.055917 463 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 10:40:12.058923 479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:40:12.060506 463 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 10:40:12.060678 463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-1/wal
uuid: "0ba17c512821475c943ab3f206c027b5"
format_stamp: "Formatted at 2026-09-25 10:40:09 on dist-test-slave-dxdd"
server_key: "0f1980c9076b839ebc4c973d5b6b9dab"
server_key_iv: "86f73d0cae81ccd35519252b123ad1ff"
server_key_version: "encryptionkey@0"
I20260925 10:40:12.061573 463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:40:12.082666 463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:40:12.086978 463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:40:12.087232 463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:40:12.088183 463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:40:12.089677 463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:40:12.089759 463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:40:12.089908 463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:40:12.089982 463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:40:12.133761 463 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.2:39099
I20260925 10:40:12.133812 591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.2:39099 every 8 connection(s)
I20260925 10:40:12.135207 463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
I20260925 10:40:12.140065 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 463
I20260925 10:40:12.140697 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.3:45801
--local_ip_for_outbound_sockets=127.23.100.3
--tserver_master_addrs=127.23.100.62:44673
--webserver_port=41447
--webserver_interface=127.23.100.3
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab
--principal=kudu/127.23.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:41457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.23.100.3
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/oryx_127.23.100.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 10:40:12 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.2@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:40:12 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.2@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 10:40:12.474849 596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:40:12.475168 596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:40:12.475340 596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:40:12.482203 596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:40:12.482358 596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:40:12.482539 596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.3
I20260925 10:40:12.491995 596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:41457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/krb5kdc/oryx_127.23.100.3.keytab
--principal=oryx/127.23.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.3:45801
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.3
--webserver_port=41447
--tserver_master_addrs=127.23.100.62:44673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.3
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:40:12.493283 596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:40:12.494714 596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:40:12.511188 602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:40:12.512166 603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:40:12.520143 605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:40:12.520998 596 server_base.cc:1034] running on GCE node
I20260925 10:40:12.521615 596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:40:12.522552 596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:40:12.523772 596 hybrid_clock.cc:648] HybridClock initialized: now 1790332812523733 us; error 28 us; skew 500 ppm
I20260925 10:40:12.524113 596 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:40:12 dist-test-slave-dxdd krb5kdc[32193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332812, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:40:12.527328 596 init.cc:377] Logged in from keytab as oryx/127.23.100.3@KRBTEST.COM (short username oryx)
I20260925 10:40:12.529548 596 webserver.cc:460] Webserver started at http://127.23.100.3:41447/ using document root <none> and password file <none>
I20260925 10:40:12.530236 596 fs_manager.cc:359] Metadata directory not provided
I20260925 10:40:12.530407 596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:40:12.535606 596 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 10:40:12.538157 612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:40:12.539541 596 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:40:12.539723 596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-2/wal
uuid: "ca8b76b6ee7c4f6aa82b94660110d1b3"
format_stamp: "Formatted at 2026-09-25 10:40:10 on dist-test-slave-dxdd"
server_key: "054581121c1ee56691a6ed69b152f13b"
server_key_iv: "334039a83706e3eb71d223318020e503"
server_key_version: "encryptionkey@0"
I20260925 10:40:12.540720 596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:40:12.554034 596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:40:12.558255 596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:40:12.558557 596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:40:12.560216 596 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:40:12.562870 596 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:40:12.562994 596 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:40:12.563127 596 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:40:12.563223 596 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:40:12.631752 596 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.3:45801
I20260925 10:40:12.631920 724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.3:45801 every 8 connection(s)
I20260925 10:40:12.633298 596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
I20260925 10:40:12.641434 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 596
Sep 25 10:40:12 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.3@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:40:12 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.3@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:40:12.732689 728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:12.643094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51706 (local address 127.23.100.62:44673)
0925 10:40:12.706836 (+ 63742us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:12.706845 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:12.706866 (+ 21us) server_negotiation.cc:407] Connection header received
0925 10:40:12.706955 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:12.706964 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:12.707068 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:12.707220 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:12.710355 (+ 3135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:12.711909 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:12.714810 (+ 2901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:12.715723 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:12.715872 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:12.722805 (+ 6933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:40:12.722817 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:40:12.722835 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:40:12.722876 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:40:12.726904 (+ 4028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:40:12.728043 (+ 1139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:40:12.728051 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:40:12.728059 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:40:12.728258 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:40:12.728734 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:40:12.728740 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:40:12.728744 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:40:12.729394 (+ 650us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:40:12.729690 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:12.731098 (+ 1408us) server_negotiation.cc:299] Negotiation successful
0925 10:40:12.731589 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63572,"thread_start_us":56443,"threads_started":1}
W20260925 10:40:13.198465 462 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.536s user 0.000s sys 0.014s
W20260925 10:40:13.200621 461 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:40:11.650176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:37587 (local address 127.23.100.62:44673)
0925 10:40:11.651244 (+ 1068us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:11.651260 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:11.651613 (+ 353us) server_negotiation.cc:407] Connection header received
0925 10:40:11.653900 (+ 2287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:11.653912 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:11.654222 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:11.654621 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:11.656182 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:11.657961 (+ 1779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:11.659907 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:11.660990 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:11.661182 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:13.200165 (+1538983us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.1:37587: BlockingRecv error: recv got EOF from 127.23.100.1:37587 (error 108)
Metrics: {"server-negotiator.queue_time_us":794,"thread_start_us":473,"threads_started":1}
I20260925 10:40:13.201355 462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:11.650536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44673 (local address 127.23.100.1:37587)
0925 10:40:11.651420 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:11.651477 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:11.653397 (+ 1920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:11.654765 (+ 1368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:11.654778 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:11.655389 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:11.655886 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:11.655903 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:11.658199 (+ 2296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:11.658208 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:11.659743 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:11.659755 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:11.661215 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:11.661220 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:11.661405 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:11.662109 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:11.662135 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:13.199737 (+1537602us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.62:44673: Server oryx/127.23.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":373,"threads_started":1}
W20260925 10:40:13.203588 459 heartbeater.cc:644] Failed to heartbeat to 127.23.100.62:44673 (0 consecutive failures): Not authorized: Failed to ping master at 127.23.100.62:44673: Client connection negotiation failed: client connection to 127.23.100.62:44673: Server oryx/127.23.100.62@KRBTEST.COM not found in Kerberos database
Sep 25 10:40:13 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.1@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:40:13 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.1@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 10:40:13.224259 461 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:40:13.205339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:39375 (local address 127.23.100.62:44673)
0925 10:40:13.205823 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:13.205832 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:13.206396 (+ 564us) server_negotiation.cc:407] Connection header received
0925 10:40:13.206719 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:13.206727 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:13.206830 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:13.206916 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:13.209453 (+ 2537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:13.211212 (+ 1759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:13.214591 (+ 3379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:13.215558 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:13.215759 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:13.223895 (+ 8136us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.1:39375: BlockingRecv error: recv got EOF from 127.23.100.1:39375 (error 108)
Metrics: {"server-negotiator.queue_time_us":325}
I20260925 10:40:13.224911 462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:13.204738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44673 (local address 127.23.100.1:39375)
0925 10:40:13.206259 (+ 1521us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:13.206285 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:13.206544 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:13.207221 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:13.207228 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:13.207897 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:13.208190 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:13.208206 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:13.211514 (+ 3308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:13.211522 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:13.214367 (+ 2845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:13.214413 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:13.215826 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:13.215837 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:13.216106 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:13.217111 (+ 1005us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:13.217144 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:13.223560 (+ 6416us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.62:44673: Server oryx/127.23.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1215}
Sep 25 10:40:13 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.1@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:40:13 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.1@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:40:13.244479 732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:13.227445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44673 (local address 127.23.100.1:50081)
0925 10:40:13.228552 (+ 1107us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:13.228580 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:13.228766 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:13.229267 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:13.229275 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:13.230113 (+ 838us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:13.230553 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:13.230570 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:13.232582 (+ 2012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:13.232590 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:13.234895 (+ 2305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:13.234912 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:13.236383 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:13.236392 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:13.236680 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:13.237620 (+ 940us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:13.237655 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:13.243969 (+ 6314us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.62:44673: Server oryx/127.23.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":898,"thread_start_us":463,"threads_started":1}
W20260925 10:40:13.244730 461 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:40:13.227541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:50081 (local address 127.23.100.62:44673)
0925 10:40:13.227768 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:13.227777 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:13.228712 (+ 935us) server_negotiation.cc:407] Connection header received
0925 10:40:13.228931 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:13.228939 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:13.229075 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:13.229225 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:13.230744 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:13.232391 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:13.235161 (+ 2770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:13.236122 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:13.236306 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:13.244302 (+ 7996us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.1:50081: BlockingRecv error: recv got EOF from 127.23.100.1:50081 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260925 10:40:13.245328 459 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 10:40:13.591799 595 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.437s user 0.006s sys 0.012s
W20260925 10:40:13.593811 594 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:40:12.137925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:50501 (local address 127.23.100.62:44673)
0925 10:40:12.138504 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:12.138510 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:12.139644 (+ 1134us) server_negotiation.cc:407] Connection header received
0925 10:40:12.145249 (+ 5605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:12.145257 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:12.145352 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:12.145457 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:12.147788 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:12.149539 (+ 1751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:12.152157 (+ 2618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:12.153161 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:12.153348 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:13.593466 (+1440118us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.2:50501: BlockingRecv error: recv got EOF from 127.23.100.2:50501 (error 108)
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":290,"threads_started":1}
I20260925 10:40:13.594822 595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:12.138293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44673 (local address 127.23.100.2:50501)
0925 10:40:12.139409 (+ 1116us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:12.139487 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:12.143096 (+ 3609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:12.145894 (+ 2798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:12.145906 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:12.146727 (+ 821us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:12.147536 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:12.147564 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:12.149724 (+ 2160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:12.149732 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:12.151994 (+ 2262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:12.152011 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:12.153430 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:12.153440 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:12.153773 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:12.154741 (+ 968us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:12.154778 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:13.593051 (+1438273us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.62:44673: Server oryx/127.23.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":664,"thread_start_us":391,"threads_started":1}
W20260925 10:40:13.595901 592 heartbeater.cc:644] Failed to heartbeat to 127.23.100.62:44673 (0 consecutive failures): Not authorized: Failed to ping master at 127.23.100.62:44673: Client connection negotiation failed: client connection to 127.23.100.62:44673: Server oryx/127.23.100.62@KRBTEST.COM not found in Kerberos database
Sep 25 10:40:13 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.2@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:40:13 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.2@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:40:13.612694 595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:13.596552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44673 (local address 127.23.100.2:60239)
0925 10:40:13.596773 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:13.596790 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:13.596880 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:13.597477 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:13.597485 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:13.598172 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:13.598533 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:13.598550 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:13.599624 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:13.599633 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:13.601959 (+ 2326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:13.601982 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:13.602861 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:13.602871 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:13.603157 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:13.607873 (+ 4716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:13.607919 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:13.612481 (+ 4562us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.62:44673: Server oryx/127.23.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":65}
W20260925 10:40:13.613200 594 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:40:13.596977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:60239 (local address 127.23.100.62:44673)
0925 10:40:13.597159 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:13.597163 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:13.597176 (+ 13us) server_negotiation.cc:407] Connection header received
0925 10:40:13.597217 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:13.597221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:13.597295 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:13.597370 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:13.598705 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:13.599462 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:13.602168 (+ 2706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:13.602642 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:13.602744 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:13.612846 (+ 10102us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.2:60239: BlockingRecv error: recv got EOF from 127.23.100.2:60239 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 25 10:40:13 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.2@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:40:13 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.2@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:40:13.622888 595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:13.613372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44673 (local address 127.23.100.2:60823)
0925 10:40:13.613758 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:13.613774 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:13.613853 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:13.614290 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:13.614293 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:13.614632 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:13.614775 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:13.614782 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:13.616742 (+ 1960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:13.616750 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:13.617826 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:13.617837 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:13.618740 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:13.618745 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:13.618935 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:13.619561 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:13.619578 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:13.622691 (+ 3113us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.62:44673: Server oryx/127.23.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249}
W20260925 10:40:13.623195 592 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 10:40:13.623481 594 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:40:13.613550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:60823 (local address 127.23.100.62:44673)
0925 10:40:13.613799 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:13.613808 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:13.613837 (+ 29us) server_negotiation.cc:407] Connection header received
0925 10:40:13.614003 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:13.614012 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:13.614131 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:13.614277 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:13.614950 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:13.616562 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:13.617967 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:13.618439 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:13.618557 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:13.623086 (+ 4529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.2:60823: BlockingRecv error: recv got EOF from 127.23.100.2:60823 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260925 10:40:13.849936 727 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.122s user 0.007s sys 0.007s
W20260925 10:40:13.851082 730 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:40:12.638327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:54377 (local address 127.23.100.62:44673)
0925 10:40:12.699269 (+ 60942us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:12.699279 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:12.706361 (+ 7082us) server_negotiation.cc:407] Connection header received
0925 10:40:12.719133 (+ 12772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:12.719138 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:12.719214 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:12.719319 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:12.721895 (+ 2576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:12.723181 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:12.725808 (+ 2627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:12.726450 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:12.726544 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:13.850875 (+1124331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.3:54377: BlockingRecv error: recv got EOF from 127.23.100.3:54377 (error 108)
Metrics: {"server-negotiator.queue_time_us":60657,"thread_start_us":69027,"threads_started":1}
I20260925 10:40:13.851693 727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:12.637530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44673 (local address 127.23.100.3:54377)
0925 10:40:12.703024 (+ 65494us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:12.703096 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:12.717922 (+ 14826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:12.719856 (+ 1934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:12.719871 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:12.720773 (+ 902us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:12.721651 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:12.721700 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:12.723379 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:12.723386 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:12.725643 (+ 2257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:12.725658 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:12.726903 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:12.726911 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:12.727211 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:12.727900 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:12.727932 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:13.850681 (+1122749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.62:44673: Server oryx/127.23.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":65056,"thread_start_us":78418,"threads_started":1}
W20260925 10:40:13.852559 725 heartbeater.cc:644] Failed to heartbeat to 127.23.100.62:44673 (0 consecutive failures): Not authorized: Failed to ping master at 127.23.100.62:44673: Client connection negotiation failed: client connection to 127.23.100.62:44673: Server oryx/127.23.100.62@KRBTEST.COM not found in Kerberos database
Sep 25 10:40:13 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.3@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:40:13 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.3@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:40:13.862033 727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:13.853139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44673 (local address 127.23.100.3:39017)
0925 10:40:13.853369 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:13.853386 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:13.853478 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:13.853833 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:13.853836 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:13.854290 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:13.854574 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:13.854587 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:13.855820 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:13.855824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:13.856761 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:13.856771 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:13.857673 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:13.857677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:13.857796 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:13.858408 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:13.858424 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:13.861753 (+ 3329us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.62:44673: Server oryx/127.23.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":69}
W20260925 10:40:13.862178 730 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:40:13.853245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:39017 (local address 127.23.100.62:44673)
0925 10:40:13.853404 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:13.853409 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:13.853425 (+ 16us) server_negotiation.cc:407] Connection header received
0925 10:40:13.853590 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:13.853594 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:13.853685 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:13.853784 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:13.854718 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:13.855699 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:13.856870 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:13.857514 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:13.857612 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:13.861939 (+ 4327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.3:39017: BlockingRecv error: recv got EOF from 127.23.100.3:39017 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 10:40:13 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.3@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:40:13 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.3@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:40:13.870430 727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:13.862625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44673 (local address 127.23.100.3:48725)
0925 10:40:13.862773 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:13.862790 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:13.862869 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:13.863161 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:13.863164 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:13.863421 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:13.863550 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:13.863556 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:13.864417 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:13.864420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:13.865383 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:13.865393 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:13.866363 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:13.866369 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:13.866569 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:13.867184 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:13.867208 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:13.870199 (+ 2991us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.62:44673: Server oryx/127.23.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260925 10:40:13.870661 730 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:40:13.862721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:48725 (local address 127.23.100.62:44673)
0925 10:40:13.862926 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:13.862930 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:13.862944 (+ 14us) server_negotiation.cc:407] Connection header received
0925 10:40:13.862980 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:13.862983 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:13.863045 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:13.863120 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:13.863658 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:13.864301 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:13.865529 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:13.866190 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:13.866338 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:13.870426 (+ 4088us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.3:48725: BlockingRecv error: recv got EOF from 127.23.100.3:48725 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
W20260925 10:40:13.870718 725 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 10:40:14 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.1@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:40:14 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.1@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:40:14.254101 733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:14.245811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44673 (local address 127.23.100.1:46481)
0925 10:40:14.246262 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:14.246278 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:14.246405 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:14.246697 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:14.246700 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:14.247043 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:14.247188 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:14.247195 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:14.248086 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:14.248089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:14.249006 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:14.249017 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:14.249641 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:14.249645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:14.249760 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:14.250405 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:14.250422 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:14.253885 (+ 3463us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.62:44673: Server oryx/127.23.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":265,"threads_started":1}
W20260925 10:40:14.254261 730 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:40:14.245916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:46481 (local address 127.23.100.62:44673)
0925 10:40:14.246167 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:14.246173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:14.246373 (+ 200us) server_negotiation.cc:407] Connection header received
0925 10:40:14.246499 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:14.246503 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:14.246582 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:14.246661 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:14.247310 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:14.247966 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:14.249126 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:14.249519 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:14.249605 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:14.254042 (+ 4437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.1:46481: BlockingRecv error: recv got EOF from 127.23.100.1:46481 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 25 10:40:14 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.2@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:40:14 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.2@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:40:14.635747 734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:14.623788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44673 (local address 127.23.100.2:42373)
0925 10:40:14.624309 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:14.624324 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:14.624476 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:14.625159 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:14.625164 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:14.625750 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:14.626017 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:14.626032 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:14.627517 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:14.627522 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:14.628819 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:14.628830 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:14.629725 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:14.629731 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:14.629934 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:14.630635 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:14.630659 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:14.635418 (+ 4759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.62:44673: Server oryx/127.23.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":286,"threads_started":1}
W20260925 10:40:14.635918 730 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:40:14.624211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:42373 (local address 127.23.100.62:44673)
0925 10:40:14.624373 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:14.624381 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:14.624411 (+ 30us) server_negotiation.cc:407] Connection header received
0925 10:40:14.624711 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:14.624720 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:14.624815 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:14.624952 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:14.626190 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:14.627328 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:14.628986 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:14.629545 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:14.629674 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:14.635686 (+ 6012us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.2:42373: BlockingRecv error: recv got EOF from 127.23.100.2:42373 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 25 10:40:14 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.3@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:40:14 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.3@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:40:14.880118 735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:14.871265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44673 (local address 127.23.100.3:48881)
0925 10:40:14.871784 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:14.871801 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:14.871917 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:14.872239 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:14.872242 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:14.872647 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:14.872786 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:14.872794 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:14.873942 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:14.873948 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:14.874956 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:14.874967 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:14.875728 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:14.875734 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:14.875898 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:14.876431 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:14.876448 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:14.879766 (+ 3318us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.62:44673: Server oryx/127.23.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":295,"threads_started":1}
W20260925 10:40:14.880218 730 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:40:14.871371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:48881 (local address 127.23.100.62:44673)
0925 10:40:14.871653 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:14.871659 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:14.871881 (+ 222us) server_negotiation.cc:407] Connection header received
0925 10:40:14.872029 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:14.872033 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:14.872108 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:14.872201 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:14.872901 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:14.873795 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:14.875099 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:14.875533 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:14.875685 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:14.879976 (+ 4291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.3:48881: BlockingRecv error: recv got EOF from 127.23.100.3:48881 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
Sep 25 10:40:15 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.1@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:40:15 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.1@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:40:15.263502 736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:15.255058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44673 (local address 127.23.100.1:47959)
0925 10:40:15.255543 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:15.255558 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:15.255663 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:15.255940 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:15.255943 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:15.256289 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:15.256440 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:15.256448 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:15.257436 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:15.257439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:15.258367 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:15.258377 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:15.259114 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:15.259118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:15.259291 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:15.259923 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:15.259971 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:15.263269 (+ 3298us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.62:44673: Server oryx/127.23.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":267,"threads_started":1}
W20260925 10:40:15.263768 730 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:40:15.255115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:47959 (local address 127.23.100.62:44673)
0925 10:40:15.255291 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:15.255295 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:15.255625 (+ 330us) server_negotiation.cc:407] Connection header received
0925 10:40:15.255753 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:15.255757 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:15.255816 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:15.255894 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:15.256555 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:15.257295 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:15.258503 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:15.258959 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:15.259082 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:15.263475 (+ 4393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.1:47959: BlockingRecv error: recv got EOF from 127.23.100.1:47959 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 25 10:40:15 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.2@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:40:15 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.2@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:40:15.646063 737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:15.636940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44673 (local address 127.23.100.2:54595)
0925 10:40:15.637412 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:15.637427 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:15.637607 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:15.638024 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:15.638028 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:15.638595 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:15.638760 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:15.638768 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:15.639756 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:15.639763 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:15.640754 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:15.640765 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:15.641542 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:15.641548 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:15.641726 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:15.642271 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:15.642288 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:15.645735 (+ 3447us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.62:44673: Server oryx/127.23.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":279,"threads_started":1}
W20260925 10:40:15.646152 730 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:40:15.637034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:54595 (local address 127.23.100.62:44673)
0925 10:40:15.637255 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:15.637262 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:15.637534 (+ 272us) server_negotiation.cc:407] Connection header received
0925 10:40:15.637766 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:15.637771 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:15.637845 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:15.637947 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:15.638897 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:15.639630 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:15.640936 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:15.641372 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:15.641528 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:15.645921 (+ 4393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.2:54595: BlockingRecv error: recv got EOF from 127.23.100.2:54595 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 25 10:40:15 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.3@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:40:15 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.3@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:40:15.889624 738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:15.880974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44673 (local address 127.23.100.3:44191)
0925 10:40:15.881527 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:15.881547 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:15.881696 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:15.882143 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:15.882148 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:15.882726 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:15.882982 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:15.882995 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:15.883935 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:15.883938 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:15.884833 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:15.884843 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:15.885473 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:15.885477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:15.885601 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:15.886154 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:15.886170 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:15.889388 (+ 3218us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.62:44673: Server oryx/127.23.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":275,"threads_started":1}
W20260925 10:40:15.889840 730 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:40:15.881152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:44191 (local address 127.23.100.62:44673)
0925 10:40:15.881376 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:15.881381 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:15.881637 (+ 256us) server_negotiation.cc:407] Connection header received
0925 10:40:15.881850 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:15.881859 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:15.881958 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:15.882081 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:15.883121 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:15.883828 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:15.884949 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:15.885348 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:15.885437 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:15.889613 (+ 4176us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.3:44191: BlockingRecv error: recv got EOF from 127.23.100.3:44191 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 25 10:40:16 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.1@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:40:16 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.1@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:40:16.273818 739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:16.264531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44673 (local address 127.23.100.1:49861)
0925 10:40:16.264998 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:16.265012 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:16.265173 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:16.265520 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:16.265524 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:16.265917 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:16.266132 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:16.266141 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:16.267397 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:16.267402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:16.268741 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:16.268753 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:16.269468 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:16.269472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:16.269605 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:16.270266 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:16.270285 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:16.273603 (+ 3318us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.62:44673: Server oryx/127.23.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":218,"threads_started":1}
W20260925 10:40:16.274024 730 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:40:16.264558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:49861 (local address 127.23.100.62:44673)
0925 10:40:16.264776 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:16.264784 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:16.265105 (+ 321us) server_negotiation.cc:407] Connection header received
0925 10:40:16.265308 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:16.265314 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:16.265390 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:16.265491 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:16.266264 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:16.267194 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:16.268870 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:16.269288 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:16.269399 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:16.273775 (+ 4376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.1:49861: BlockingRecv error: recv got EOF from 127.23.100.1:49861 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 25 10:40:16 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.2@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:40:16 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.2@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:40:16.656064 740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:16.647185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44673 (local address 127.23.100.2:50535)
0925 10:40:16.647714 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:16.647732 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:16.647855 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:16.648161 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:16.648164 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:16.648490 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:16.648675 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:16.648682 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:16.649634 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:16.649637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:16.650484 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:16.650493 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:16.651217 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:16.651221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:16.651342 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:16.651898 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:16.651915 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:16.655728 (+ 3813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.62:44673: Server oryx/127.23.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":258,"threads_started":1}
W20260925 10:40:16.656204 730 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:40:16.647267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:50535 (local address 127.23.100.62:44673)
0925 10:40:16.647513 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:16.647519 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:16.647814 (+ 295us) server_negotiation.cc:407] Connection header received
0925 10:40:16.647957 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:16.647960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:16.648021 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:16.648101 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:16.648805 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:16.649524 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:16.650611 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:16.651067 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:16.651207 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:16.655968 (+ 4761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.2:50535: BlockingRecv error: recv got EOF from 127.23.100.2:50535 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 25 10:40:16 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.3@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:40:16 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.3@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:40:16.899224 741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:16.890547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44673 (local address 127.23.100.3:43645)
0925 10:40:16.891092 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:16.891111 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:16.891248 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:16.891621 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:16.891624 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:16.891951 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:16.892120 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:16.892128 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:16.893055 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:16.893058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:16.894497 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:16.894509 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:16.895136 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:16.895140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:16.895262 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:16.895784 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:16.895812 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:16.898988 (+ 3176us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.62:44673: Server oryx/127.23.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":249,"threads_started":1}
W20260925 10:40:16.899376 730 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:40:16.890580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:43645 (local address 127.23.100.62:44673)
0925 10:40:16.890766 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:16.890775 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:16.891176 (+ 401us) server_negotiation.cc:407] Connection header received
0925 10:40:16.891393 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:16.891400 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:16.891490 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:16.891577 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:16.892241 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:16.892932 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:16.894642 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:16.895019 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:16.895113 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:16.899160 (+ 4047us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.3:43645: BlockingRecv error: recv got EOF from 127.23.100.3:43645 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 25 10:40:17 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.1@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:40:17 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.1@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:40:17.283892 742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:17.274829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44673 (local address 127.23.100.1:43931)
0925 10:40:17.275248 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:17.275263 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:17.275368 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:17.275656 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:17.275658 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:17.276019 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:17.276169 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:17.276176 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:17.277156 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:17.277162 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:17.278751 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:17.278765 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:17.279412 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:17.279416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:17.279529 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:17.280117 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:17.280133 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:17.283536 (+ 3403us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.62:44673: Server oryx/127.23.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":241,"threads_started":1}
W20260925 10:40:17.283994 730 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:40:17.274881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:43931 (local address 127.23.100.62:44673)
0925 10:40:17.275120 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:17.275126 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:17.275344 (+ 218us) server_negotiation.cc:407] Connection header received
0925 10:40:17.275463 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:17.275467 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:17.275526 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:17.275603 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:17.276284 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:17.277011 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:17.278877 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:17.279293 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:17.279387 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:17.283739 (+ 4352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.1:43931: BlockingRecv error: recv got EOF from 127.23.100.1:43931 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 25 10:40:17 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.2@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:40:17 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.2@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:40:17.665817 743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:17.656916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44673 (local address 127.23.100.2:60335)
0925 10:40:17.657374 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:17.657388 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:17.657537 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:17.657878 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:17.657882 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:17.658353 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:17.658522 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:17.658531 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:17.659508 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:17.659516 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:17.660517 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:17.660528 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:17.661292 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:17.661297 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:17.661491 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:17.662107 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:17.662124 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:17.665451 (+ 3327us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.62:44673: Server oryx/127.23.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":263,"threads_started":1}
W20260925 10:40:17.665879 730 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:40:17.656990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:60335 (local address 127.23.100.62:44673)
0925 10:40:17.657183 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:17.657191 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:17.657500 (+ 309us) server_negotiation.cc:407] Connection header received
0925 10:40:17.657665 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:17.657670 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:17.657736 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:17.657837 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:17.658664 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:17.659375 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:17.660662 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:17.661140 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:17.661271 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:17.665649 (+ 4378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.2:60335: BlockingRecv error: recv got EOF from 127.23.100.2:60335 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 25 10:40:17 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.3@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:40:17 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.3@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:40:17.909394 744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:17.900333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44673 (local address 127.23.100.3:41199)
0925 10:40:17.900863 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:17.900879 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:17.901041 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:17.901427 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:17.901432 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:17.901832 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:17.902007 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:17.902015 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:17.903241 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:17.903247 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:17.904285 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:17.904296 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:17.904963 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:17.904967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:17.905098 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:17.905646 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:17.905662 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:17.909162 (+ 3500us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.62:44673: Server oryx/127.23.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":255,"threads_started":1}
W20260925 10:40:17.909632 730 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:40:17.900368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:41199 (local address 127.23.100.62:44673)
0925 10:40:17.900568 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:17.900576 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:17.901017 (+ 441us) server_negotiation.cc:407] Connection header received
0925 10:40:17.901199 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:17.901204 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:17.901280 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:17.901399 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:17.902152 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:17.903034 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:17.904435 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:17.904840 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:17.904938 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:17.909354 (+ 4416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.3:41199: BlockingRecv error: recv got EOF from 127.23.100.3:41199 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 25 10:40:18 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.1@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:40:18 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.1@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:40:18.293764 745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:18.285032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44673 (local address 127.23.100.1:40927)
0925 10:40:18.285529 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:18.285549 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:18.285692 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:18.286083 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:18.286087 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:18.286621 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:18.286842 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:18.286854 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:18.287935 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:18.287939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:18.288959 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:18.288971 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:18.289717 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:18.289721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:18.289837 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:18.290479 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:18.290496 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:18.293561 (+ 3065us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.62:44673: Server oryx/127.23.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":263,"threads_started":1}
W20260925 10:40:18.293911 730 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:40:18.285068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:40927 (local address 127.23.100.62:44673)
0925 10:40:18.285284 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:18.285293 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:18.285645 (+ 352us) server_negotiation.cc:407] Connection header received
0925 10:40:18.285831 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:18.285838 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:18.285928 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:18.286011 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:18.286990 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:18.287821 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:18.289092 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:18.289572 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:18.289698 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:18.293725 (+ 4027us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.1:40927: BlockingRecv error: recv got EOF from 127.23.100.1:40927 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 25 10:40:18 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.2@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:40:18 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.2@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:40:18.681088 746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:18.666786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44673 (local address 127.23.100.2:38495)
0925 10:40:18.667256 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:18.667272 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:18.667428 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:18.667783 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:18.667788 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:18.668199 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:18.668344 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:18.668353 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:18.669387 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:18.669393 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:18.670288 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:18.670298 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:18.671023 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:18.671028 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:18.671206 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:18.677567 (+ 6361us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:18.677586 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:18.680862 (+ 3276us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.62:44673: Server oryx/127.23.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":226,"threads_started":1}
W20260925 10:40:18.681293 730 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:40:18.667003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:38495 (local address 127.23.100.62:44673)
0925 10:40:18.667196 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:18.667202 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:18.667344 (+ 142us) server_negotiation.cc:407] Connection header received
0925 10:40:18.667550 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:18.667575 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:18.667646 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:18.667747 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:18.668484 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:18.669252 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:18.670441 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:18.670881 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:18.671015 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:18.681052 (+ 10037us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.2:38495: BlockingRecv error: recv got EOF from 127.23.100.2:38495 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 25 10:40:18 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.3@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:40:18 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.3@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:40:18.919245 747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:18.910443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44673 (local address 127.23.100.3:55223)
0925 10:40:18.910933 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:18.910949 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:18.911135 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:18.911496 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:18.911501 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:18.911926 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:18.912074 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:18.912082 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:18.913075 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:18.913083 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:18.913980 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:18.913991 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:18.914771 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:18.914777 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:18.914949 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:18.915567 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:18.915584 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:18.918929 (+ 3345us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.62:44673: Server oryx/127.23.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":240,"threads_started":1}
W20260925 10:40:18.919325 730 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:40:18.910492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:55223 (local address 127.23.100.62:44673)
0925 10:40:18.910693 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:18.910701 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:18.911057 (+ 356us) server_negotiation.cc:407] Connection header received
0925 10:40:18.911287 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:18.911293 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:18.911361 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:18.911472 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:18.912220 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:18.912948 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:18.914121 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:18.914590 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:18.914744 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:18.919106 (+ 4362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.3:55223: BlockingRecv error: recv got EOF from 127.23.100.3:55223 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 25 10:40:19 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.1@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:40:19 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.1@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:40:19.302634 748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:19.294556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44673 (local address 127.23.100.1:36433)
0925 10:40:19.295084 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:19.295101 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:19.295204 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:19.295476 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:19.295480 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:19.295868 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:19.296021 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:19.296028 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:19.296899 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:19.296902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:19.297790 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:19.297800 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:19.298433 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:19.298437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:19.298550 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:19.299070 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:19.299086 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:19.302291 (+ 3205us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.62:44673: Server oryx/127.23.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":280,"threads_started":1}
W20260925 10:40:19.302778 730 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:40:19.294651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:36433 (local address 127.23.100.62:44673)
0925 10:40:19.294874 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:19.294878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:19.295170 (+ 292us) server_negotiation.cc:407] Connection header received
0925 10:40:19.295297 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:19.295300 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:19.295354 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:19.295435 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:19.296141 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:19.296799 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:19.297899 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:19.298297 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:19.298393 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:19.302516 (+ 4123us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.1:36433: BlockingRecv error: recv got EOF from 127.23.100.1:36433 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 25 10:40:19 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.2@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:40:19 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.2@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:40:19.690876 749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:19.682004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44673 (local address 127.23.100.2:57195)
0925 10:40:19.682516 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:19.682531 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:19.682687 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:19.683050 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:19.683055 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:19.683470 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:19.683615 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:19.683623 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:19.684596 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:19.684602 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:19.685510 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:19.685520 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:19.686255 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:19.686261 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:19.686468 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:19.687055 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:19.687073 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:19.690665 (+ 3592us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.62:44673: Server oryx/127.23.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":287,"threads_started":1}
W20260925 10:40:19.690997 730 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:40:19.682081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:57195 (local address 127.23.100.62:44673)
0925 10:40:19.682305 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:19.682330 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:19.682619 (+ 289us) server_negotiation.cc:407] Connection header received
0925 10:40:19.682819 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:19.682824 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:19.682890 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:19.682988 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:19.683760 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:19.684444 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:19.685655 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:19.686097 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:19.686240 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:19.690823 (+ 4583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.2:57195: BlockingRecv error: recv got EOF from 127.23.100.2:57195 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 25 10:40:19 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.3@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:40:19 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.3@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:40:19.929309 750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:19.920259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44673 (local address 127.23.100.3:56189)
0925 10:40:19.920707 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:19.920722 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:19.920834 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:19.921138 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:19.921142 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:19.921517 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:19.921671 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:19.921679 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:19.922739 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:19.922746 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:19.924161 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:19.924172 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:19.924796 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:19.924800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:19.924918 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:19.925679 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:19.925705 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:19.929075 (+ 3370us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.62:44673: Server oryx/127.23.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":266,"threads_started":1}
W20260925 10:40:19.929507 730 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:40:19.920361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:56189 (local address 127.23.100.62:44673)
0925 10:40:19.920650 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:19.920656 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:19.920802 (+ 146us) server_negotiation.cc:407] Connection header received
0925 10:40:19.920935 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:19.920938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:19.921011 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:19.921102 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:19.921786 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:19.922580 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:19.924276 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:19.924658 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:19.924750 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:19.929254 (+ 4504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.3:56189: BlockingRecv error: recv got EOF from 127.23.100.3:56189 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Sep 25 10:40:20 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.1@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:40:20 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.1@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:40:20.312592 751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:20.303904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44673 (local address 127.23.100.1:39645)
0925 10:40:20.304498 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:20.304513 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:20.304620 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:20.304913 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:20.304916 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:20.305307 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:20.305478 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:20.305486 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:20.306467 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:20.306470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:20.307363 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:20.307373 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:20.308016 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:20.308020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:20.308212 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:20.308887 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:20.308905 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:20.312335 (+ 3430us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.62:44673: Server oryx/127.23.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":278,"threads_started":1}
W20260925 10:40:20.312717 730 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:40:20.303986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:39645 (local address 127.23.100.62:44673)
0925 10:40:20.304168 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:20.304172 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:20.304583 (+ 411us) server_negotiation.cc:407] Connection header received
0925 10:40:20.304709 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:20.304712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:20.304772 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:20.304870 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:20.305608 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:20.306297 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:20.307477 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:20.307900 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:20.307994 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:20.312527 (+ 4533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.1:39645: BlockingRecv error: recv got EOF from 127.23.100.1:39645 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 25 10:40:20 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.2@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:40:20 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.2@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:40:20.700836 752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:20.691729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44673 (local address 127.23.100.2:36697)
0925 10:40:20.692323 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:20.692338 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:20.692506 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:20.692846 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:20.692850 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:20.693276 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:20.693460 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:20.693470 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:20.694556 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:20.694563 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:20.695508 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:20.695519 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:20.696214 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:20.696219 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:20.696404 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:20.697133 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:20.697150 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:20.700583 (+ 3433us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.62:44673: Server oryx/127.23.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":282,"threads_started":1}
W20260925 10:40:20.701007 730 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:40:20.691785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:36697 (local address 127.23.100.62:44673)
0925 10:40:20.692023 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:20.692032 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:20.692431 (+ 399us) server_negotiation.cc:407] Connection header received
0925 10:40:20.692632 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:20.692637 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:20.692705 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:20.692808 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:20.693607 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:20.694431 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:20.695651 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:20.696058 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:20.696187 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:20.700750 (+ 4563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.2:36697: BlockingRecv error: recv got EOF from 127.23.100.2:36697 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 25 10:40:20 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.3@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:40:20 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.3@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:40:20.938819 753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:20.930232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44673 (local address 127.23.100.3:40907)
0925 10:40:20.930894 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:20.930909 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:20.931031 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:20.931324 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:20.931327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:20.931823 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:20.932081 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:20.932113 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:20.933188 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:20.933191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:20.934091 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:20.934101 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:20.934728 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:20.934732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:20.934863 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:20.935396 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:20.935412 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:20.938606 (+ 3194us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.62:44673: Server oryx/127.23.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":301,"threads_started":1}
W20260925 10:40:20.938961 730 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:40:20.930337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:40907 (local address 127.23.100.62:44673)
0925 10:40:20.930496 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:20.930500 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:20.930980 (+ 480us) server_negotiation.cc:407] Connection header received
0925 10:40:20.931142 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:20.931145 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:20.931202 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:20.931281 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:20.932245 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:20.933059 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:20.934202 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:20.934611 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:20.934697 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:20.938773 (+ 4076us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.3:40907: BlockingRecv error: recv got EOF from 127.23.100.3:40907 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 25 10:40:21 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.1@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:40:21 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.1@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:40:21.322474 754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:21.313610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44673 (local address 127.23.100.1:34017)
0925 10:40:21.314146 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:21.314162 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:21.314270 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:21.314572 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:21.314575 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:21.314947 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:21.315132 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:21.315140 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:21.316105 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:21.316109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:21.316970 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:21.316979 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:21.317621 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:21.317625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:21.317734 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:21.318361 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:21.318387 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:21.322175 (+ 3788us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.62:44673: Server oryx/127.23.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":276,"threads_started":1}
W20260925 10:40:21.322656 730 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:40:21.313766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:34017 (local address 127.23.100.62:44673)
0925 10:40:21.313941 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:21.313945 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:21.314233 (+ 288us) server_negotiation.cc:407] Connection header received
0925 10:40:21.314385 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:21.314389 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:21.314446 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:21.314523 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:21.315248 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:21.315966 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:21.317081 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:21.317478 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:21.317564 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:21.322421 (+ 4857us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.1:34017: BlockingRecv error: recv got EOF from 127.23.100.1:34017 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 25 10:40:21 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.2@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:40:21 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.2@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:40:21.710685 755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:21.701727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44673 (local address 127.23.100.2:48023)
0925 10:40:21.702247 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:21.702262 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:21.702457 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:21.702794 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:21.702798 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:21.703218 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:21.703386 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:21.703394 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:21.704536 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:21.704542 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:21.705477 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:21.705487 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:21.706252 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:21.706259 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:21.706454 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:21.707164 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:21.707182 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:21.710443 (+ 3261us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.62:44673: Server oryx/127.23.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":256,"threads_started":1}
W20260925 10:40:21.710909 730 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:40:21.701825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:48023 (local address 127.23.100.62:44673)
0925 10:40:21.702070 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:21.702077 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:21.702362 (+ 285us) server_negotiation.cc:407] Connection header received
0925 10:40:21.702582 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:21.702588 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:21.702654 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:21.702757 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:21.703537 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:21.704385 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:21.705617 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:21.706060 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:21.706191 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:21.710656 (+ 4465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.2:48023: BlockingRecv error: recv got EOF from 127.23.100.2:48023 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 25 10:40:21 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.3@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:40:21 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.3@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:40:21.948525 756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:21.939723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44673 (local address 127.23.100.3:47083)
0925 10:40:21.940318 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:21.940338 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:21.940481 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:21.940741 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:21.940744 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:21.941166 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:21.941316 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:21.941323 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:21.942438 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:21.942441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:21.943365 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:21.943374 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:21.944159 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:21.944163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:21.944272 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:21.944920 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:21.944937 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:21.948295 (+ 3358us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.62:44673: Server oryx/127.23.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":318,"threads_started":1}
W20260925 10:40:21.948683 730 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:40:21.939823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:47083 (local address 127.23.100.62:44673)
0925 10:40:21.940069 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:21.940077 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:21.940427 (+ 350us) server_negotiation.cc:407] Connection header received
0925 10:40:21.940577 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:21.940580 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:21.940639 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:21.940716 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:21.941474 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:21.942285 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:21.943488 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:21.944012 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:21.944109 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:21.948468 (+ 4359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.3:47083: BlockingRecv error: recv got EOF from 127.23.100.3:47083 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 25 10:40:22 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.1@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:40:22 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.1@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:40:22.332206 757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:22.323376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44673 (local address 127.23.100.1:54411)
0925 10:40:22.323851 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:22.323866 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:22.324016 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:22.324364 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:22.324369 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:22.324796 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:22.324946 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:22.324955 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:22.325975 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:22.325980 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:22.327080 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:22.327091 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:22.327809 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:22.327815 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:22.328001 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:22.328603 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:22.328620 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:22.331990 (+ 3370us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.62:44673: Server oryx/127.23.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":260,"threads_started":1}
W20260925 10:40:22.332362 730 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:40:22.323471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:54411 (local address 127.23.100.62:44673)
0925 10:40:22.323712 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:22.323719 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:22.323949 (+ 230us) server_negotiation.cc:407] Connection header received
0925 10:40:22.324146 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:22.324155 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:22.324233 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:22.324333 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:22.325098 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:22.325831 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:22.327223 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:22.327652 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:22.327799 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:22.332153 (+ 4354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.1:54411: BlockingRecv error: recv got EOF from 127.23.100.1:54411 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 25 10:40:22 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.2@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:40:22 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.2@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:40:22.720176 758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:22.711763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44673 (local address 127.23.100.2:41787)
0925 10:40:22.712234 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:22.712249 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:22.712349 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:22.712659 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:22.712663 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:22.713105 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:22.713244 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:22.713253 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:22.714358 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:22.714362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:22.715297 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:22.715307 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:22.715928 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:22.715932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:22.716064 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:22.716587 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:22.716603 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:22.719961 (+ 3358us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.62:44673: Server oryx/127.23.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":245,"threads_started":1}
W20260925 10:40:22.720392 730 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:40:22.711800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:41787 (local address 127.23.100.62:44673)
0925 10:40:22.711992 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:22.711998 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:22.712330 (+ 332us) server_negotiation.cc:407] Connection header received
0925 10:40:22.712431 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:22.712436 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:22.712508 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:22.712623 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:22.713448 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:22.714211 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:22.715413 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:22.715799 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:22.715888 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:22.720137 (+ 4249us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.2:41787: BlockingRecv error: recv got EOF from 127.23.100.2:41787 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 25 10:40:22 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.3@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:40:22 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.3@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:40:22.957816 759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:22.949492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44673 (local address 127.23.100.3:58649)
0925 10:40:22.949959 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:22.949973 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:22.950078 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:22.950365 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:22.950368 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:22.950765 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:22.950895 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:22.950903 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:22.951833 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:22.951836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:22.952774 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:22.952784 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:22.953402 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:22.953406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:22.953525 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:22.954086 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:22.954102 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:22.957463 (+ 3361us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.62:44673: Server oryx/127.23.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":259,"threads_started":1}
W20260925 10:40:22.957957 730 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:40:22.949528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:58649 (local address 127.23.100.62:44673)
0925 10:40:22.949697 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:22.949701 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:22.950041 (+ 340us) server_negotiation.cc:407] Connection header received
0925 10:40:22.950172 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:22.950176 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:22.950231 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:22.950354 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:22.951006 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:22.951728 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:22.952883 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:22.953284 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:22.953371 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:22.957696 (+ 4325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.3:58649: BlockingRecv error: recv got EOF from 127.23.100.3:58649 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 25 10:40:23 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.1@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:40:23 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.1@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:40:23.341889 760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:23.333168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44673 (local address 127.23.100.1:40567)
0925 10:40:23.333630 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:23.333645 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:23.333749 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:23.334147 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:23.334152 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:23.334571 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:23.334721 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:23.334729 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:23.336039 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:23.336043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:23.336981 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:23.336991 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:23.337627 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:23.337631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:23.337739 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:23.338295 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:23.338335 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:23.341669 (+ 3334us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.62:44673: Server oryx/127.23.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":247,"threads_started":1}
W20260925 10:40:23.342090 730 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:40:23.333234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:40567 (local address 127.23.100.62:44673)
0925 10:40:23.333426 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:23.333430 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:23.333719 (+ 289us) server_negotiation.cc:407] Connection header received
0925 10:40:23.333897 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:23.333904 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:23.333986 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:23.334103 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:23.334909 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:23.335893 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:23.337103 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:23.337505 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:23.337594 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:23.341857 (+ 4263us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.1:40567: BlockingRecv error: recv got EOF from 127.23.100.1:40567 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 25 10:40:23 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.2@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:40:23 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.2@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:40:23.737061 761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:23.721246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44673 (local address 127.23.100.2:36459)
0925 10:40:23.721794 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:23.721815 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:23.721977 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:23.722294 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:23.722297 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:23.722678 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:23.722825 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:23.722833 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:23.723755 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:23.723758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:23.724712 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:23.724722 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:23.725369 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:23.725376 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:23.725557 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:23.733212 (+ 7655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:23.733230 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:23.736727 (+ 3497us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.62:44673: Server oryx/127.23.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":301,"threads_started":1}
W20260925 10:40:23.737217 730 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:40:23.721350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:36459 (local address 127.23.100.62:44673)
0925 10:40:23.721609 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:23.721614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:23.721911 (+ 297us) server_negotiation.cc:407] Connection header received
0925 10:40:23.722110 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:23.722113 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:23.722173 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:23.722250 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:23.722969 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:23.723642 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:23.724828 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:23.725210 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:23.725299 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:23.736978 (+ 11679us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.2:36459: BlockingRecv error: recv got EOF from 127.23.100.2:36459 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 25 10:40:23 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.3@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:40:23 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.3@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:40:23.968039 762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:23.958695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44673 (local address 127.23.100.3:58909)
0925 10:40:23.959272 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:23.959286 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:23.959433 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:23.959792 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:23.959796 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:23.960213 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:23.960363 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:23.960371 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:23.961369 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:23.961377 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:23.962424 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:23.962434 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:23.963166 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:23.963171 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:23.963334 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:23.964037 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:23.964053 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:23.967679 (+ 3626us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.62:44673: Server oryx/127.23.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":357,"threads_started":1}
W20260925 10:40:23.968185 730 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:40:23.958894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:58909 (local address 127.23.100.62:44673)
0925 10:40:23.959147 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:23.959153 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:23.959365 (+ 212us) server_negotiation.cc:407] Connection header received
0925 10:40:23.959578 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:23.959583 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:23.959659 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:23.959758 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:23.960505 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:23.961238 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:23.962569 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:23.963019 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:23.963155 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:23.967895 (+ 4740us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.3:58909: BlockingRecv error: recv got EOF from 127.23.100.3:58909 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Sep 25 10:40:24 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.1@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:40:24 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.1@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:40:24.351845 763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:24.342843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44673 (local address 127.23.100.1:34415)
0925 10:40:24.343368 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:24.343387 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:24.343547 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:24.343927 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:24.343931 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:24.344513 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:24.344742 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:24.344756 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:24.345788 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:24.345792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:24.346717 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:24.346727 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:24.347379 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:24.347383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:24.347511 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:24.348148 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:24.348165 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:24.351633 (+ 3468us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.62:44673: Server oryx/127.23.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":253,"threads_started":1}
W20260925 10:40:24.352048 730 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:40:24.342903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:34415 (local address 127.23.100.62:44673)
0925 10:40:24.343133 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:24.343141 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:24.343491 (+ 350us) server_negotiation.cc:407] Connection header received
0925 10:40:24.343691 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:24.343696 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:24.343774 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:24.343868 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:24.344881 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:24.345675 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:24.346828 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:24.347254 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:24.347344 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:24.351801 (+ 4457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.1:34415: BlockingRecv error: recv got EOF from 127.23.100.1:34415 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 25 10:40:24 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.2@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:40:24 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.2@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:40:24.747099 764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:24.738231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44673 (local address 127.23.100.2:44641)
0925 10:40:24.738766 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:24.738783 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:24.738893 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:24.739202 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:24.739205 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:24.739609 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:24.739780 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:24.739788 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:24.740796 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:24.740800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:24.741689 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:24.741699 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:24.742319 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:24.742323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:24.742492 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:24.743046 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:24.743062 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:24.746880 (+ 3818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.62:44673: Server oryx/127.23.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":322,"threads_started":1}
W20260925 10:40:24.747265 730 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:40:24.738359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:44641 (local address 127.23.100.62:44673)
0925 10:40:24.738510 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:24.738515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:24.738853 (+ 338us) server_negotiation.cc:407] Connection header received
0925 10:40:24.739002 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:24.739005 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:24.739088 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:24.739161 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:24.739921 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:24.740690 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:24.741801 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:24.742183 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:24.742270 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:24.747033 (+ 4763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.2:44641: BlockingRecv error: recv got EOF from 127.23.100.2:44641 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 25 10:40:24 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.3@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:40:24 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.3@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:40:24.978021 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:24.969010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44673 (local address 127.23.100.3:56739)
0925 10:40:24.969716 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:24.969733 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:24.969865 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:24.970242 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:24.970245 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:24.970718 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:24.970903 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:24.970911 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:24.971903 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:24.971911 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:24.972827 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:24.972838 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:24.973545 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:24.973551 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:24.973731 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:24.974406 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:24.974424 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:24.977788 (+ 3364us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.62:44673: Server oryx/127.23.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":490,"threads_started":1}
W20260925 10:40:24.978175 730 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:40:24.969079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:56739 (local address 127.23.100.62:44673)
0925 10:40:24.969254 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:24.969262 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:24.969838 (+ 576us) server_negotiation.cc:407] Connection header received
0925 10:40:24.970044 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:24.970048 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:24.970110 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:24.970194 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:24.971052 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:24.971772 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:24.972981 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:24.973380 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:24.973527 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:24.977959 (+ 4432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.3:56739: BlockingRecv error: recv got EOF from 127.23.100.3:56739 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 25 10:40:25 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.1@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:40:25 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.1@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:40:25.361739 766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:25.352798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44673 (local address 127.23.100.1:34985)
0925 10:40:25.353319 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:25.353337 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:25.353450 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:25.353731 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:25.353734 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:25.354112 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:25.354267 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:25.354275 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:25.355293 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:25.355297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:25.356402 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:25.356420 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:25.357293 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:25.357301 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:25.357487 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:25.358077 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:25.358093 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:25.361390 (+ 3297us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.62:44673: Server oryx/127.23.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":303,"threads_started":1}
W20260925 10:40:25.361807 730 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:40:25.352874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:34985 (local address 127.23.100.62:44673)
0925 10:40:25.353132 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:25.353138 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:25.353422 (+ 284us) server_negotiation.cc:407] Connection header received
0925 10:40:25.353546 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:25.353550 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:25.353607 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:25.353686 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:25.354438 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:25.355181 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:25.356578 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:25.357116 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:25.357259 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:25.361577 (+ 4318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.1:34985: BlockingRecv error: recv got EOF from 127.23.100.1:34985 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 25 10:40:25 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.2@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:40:25 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.2@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:40:25.757066 767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:25.747965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44673 (local address 127.23.100.2:37445)
0925 10:40:25.748485 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:25.748510 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:25.748618 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:25.748936 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:25.748938 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:25.749309 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:25.749468 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:25.749476 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:25.750445 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:25.750448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:25.751420 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:25.751433 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:25.752362 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:25.752368 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:25.752644 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:25.753311 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:25.753328 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:25.756714 (+ 3386us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.62:44673: Server oryx/127.23.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":292,"threads_started":1}
W20260925 10:40:25.757166 730 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:40:25.748108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:37445 (local address 127.23.100.62:44673)
0925 10:40:25.748282 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:25.748286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:25.748583 (+ 297us) server_negotiation.cc:407] Connection header received
0925 10:40:25.748747 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:25.748752 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:25.748814 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:25.748891 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:25.749598 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:25.750334 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:25.751556 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:25.752211 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:25.752317 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:25.756914 (+ 4597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.2:37445: BlockingRecv error: recv got EOF from 127.23.100.2:37445 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 25 10:40:25 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.3@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:40:25 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.3@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:40:25.987591 768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:25.979005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44673 (local address 127.23.100.3:35133)
0925 10:40:25.979492 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:25.979509 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:25.979625 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:25.979933 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:25.979937 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:25.980352 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:25.980523 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:25.980531 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:25.981492 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:25.981495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:25.982434 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:25.982445 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:25.983207 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:25.983212 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:25.983350 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:25.984054 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:25.984072 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:25.987353 (+ 3281us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.62:44673: Server oryx/127.23.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":261,"threads_started":1}
W20260925 10:40:25.987766 730 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:40:25.979094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:35133 (local address 127.23.100.62:44673)
0925 10:40:25.979334 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:25.979340 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:25.979584 (+ 244us) server_negotiation.cc:407] Connection header received
0925 10:40:25.979724 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:25.979728 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:25.979801 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:25.979880 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:25.980640 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:25.981367 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:25.982546 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:25.982930 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:25.983044 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:25.987540 (+ 4496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.3:35133: BlockingRecv error: recv got EOF from 127.23.100.3:35133 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Sep 25 10:40:26 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.1@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:40:26 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.1@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:40:26.371367 769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:26.362708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44673 (local address 127.23.100.1:60567)
0925 10:40:26.363166 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:26.363179 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:26.363326 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:26.363654 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:26.363659 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:26.364049 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:26.364182 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:26.364190 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:26.365264 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:26.365271 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:26.366253 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:26.366265 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:26.366978 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:26.366984 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:26.367161 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:26.367741 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:26.367757 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:26.371155 (+ 3398us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.62:44673: Server oryx/127.23.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":260,"threads_started":1}
W20260925 10:40:26.371563 730 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:40:26.362779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:60567 (local address 127.23.100.62:44673)
0925 10:40:26.362983 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:26.362991 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:26.363259 (+ 268us) server_negotiation.cc:407] Connection header received
0925 10:40:26.363442 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:26.363447 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:26.363518 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:26.363617 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:26.364319 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:26.365113 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:26.366420 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:26.366831 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:26.366959 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:26.371339 (+ 4380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.1:60567: BlockingRecv error: recv got EOF from 127.23.100.1:60567 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 25 10:40:26 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.2@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:40:26 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.2@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:40:26.766368 770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:26.757992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44673 (local address 127.23.100.2:58517)
0925 10:40:26.758539 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:26.758555 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:26.758706 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:26.759002 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:26.759005 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:26.759364 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:26.759510 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:26.759518 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:26.760490 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:26.760494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:26.761385 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:26.761395 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:26.762018 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:26.762022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:26.762136 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:26.762787 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:26.762804 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:26.766081 (+ 3277us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.62:44673: Server oryx/127.23.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":321,"threads_started":1}
W20260925 10:40:26.766511 730 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:40:26.758147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:58517 (local address 127.23.100.62:44673)
0925 10:40:26.758417 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:26.758422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:26.758683 (+ 261us) server_negotiation.cc:407] Connection header received
0925 10:40:26.758805 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:26.758808 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:26.758866 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:26.758943 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:26.759620 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:26.760376 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:26.761505 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:26.761886 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:26.762018 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:26.766237 (+ 4219us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.2:58517: BlockingRecv error: recv got EOF from 127.23.100.2:58517 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 25 10:40:26 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.3@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:40:26 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.3@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:40:26.997262 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:26.988670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44673 (local address 127.23.100.3:50477)
0925 10:40:26.989176 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:26.989192 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:26.989303 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:26.989595 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:26.989598 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:26.990072 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:26.990219 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:26.990227 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:26.991238 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:26.991242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:26.992098 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:26.992108 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:26.992736 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:26.992740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:26.992849 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:26.993376 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:26.993392 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:26.997016 (+ 3624us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.62:44673: Server oryx/127.23.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":287,"threads_started":1}
W20260925 10:40:26.997454 730 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:40:26.988746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:50477 (local address 127.23.100.62:44673)
0925 10:40:26.988910 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:26.988915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:26.989265 (+ 350us) server_negotiation.cc:407] Connection header received
0925 10:40:26.989402 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:26.989405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:26.989461 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:26.989554 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:26.990373 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:26.991122 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:26.992219 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:26.992621 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:26.992705 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:26.997220 (+ 4515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.3:50477: BlockingRecv error: recv got EOF from 127.23.100.3:50477 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 25 10:40:27 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.1@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:40:27 dist-test-slave-dxdd krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.1@KRBTEST.COM for oryx/127.23.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:40:27.381177 772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:27.372559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44673 (local address 127.23.100.1:53037)
0925 10:40:27.373042 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:27.373057 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:27.373159 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:27.373438 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:27.373440 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:27.373773 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:27.373939 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:27.373946 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:27.374922 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:27.374925 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:27.376250 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:27.376279 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:27.376993 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:27.376997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:27.377115 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:27.377668 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:27.377683 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:27.380941 (+ 3258us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.62:44673: Server oryx/127.23.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":244,"threads_started":1}
W20260925 10:40:27.381433 730 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:40:27.372701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:53037 (local address 127.23.100.62:44673)
0925 10:40:27.372978 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:27.372991 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:27.373131 (+ 140us) server_negotiation.cc:407] Connection header received
0925 10:40:27.373252 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:27.373255 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:27.373312 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:27.373390 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:27.374054 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:27.374806 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:27.376386 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:27.376874 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:27.376964 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:27.381139 (+ 4175us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.1:53037: BlockingRecv error: recv got EOF from 127.23.100.1:53037 (error 108)
Metrics: {"server-negotiator.queue_time_us":172}
I20260925 10:40:27.645783 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 312
I20260925 10:40:27.658771 774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:27.646951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47332 (local address 127.23.100.1:45675)
0925 10:40:27.647689 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:27.647698 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:27.647727 (+ 29us) server_negotiation.cc:407] Connection header received
0925 10:40:27.647797 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:27.647806 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:27.648105 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:27.648323 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:27.649688 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:27.651010 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:27.652228 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:27.652822 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:27.652935 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:27.655298 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:40:27.655306 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:40:27.655318 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:40:27.655351 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:40:27.658143 (+ 2792us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.23.100.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 10:40:27.658503 (+ 360us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47332: Request ticket server kudu/127.23.100.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":396,"threads_started":1}
W20260925 10:40:27.658794 23952 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 312: Not authorized: Client connection negotiation failed: client connection to 127.23.100.1:45675: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.23.100.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260925 10:40:27.658875 774 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47332: Request ticket server kudu/127.23.100.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 10:40:27.659000 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 312
I20260925 10:40:27.675625 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 463
I20260925 10:40:27.687601 775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:27.676589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44680 (local address 127.23.100.2:39099)
0925 10:40:27.677163 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:27.677169 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:27.677193 (+ 24us) server_negotiation.cc:407] Connection header received
0925 10:40:27.677253 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:27.677259 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:27.677490 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:27.677647 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:27.679321 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:27.680312 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:27.681464 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:27.681964 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:27.682062 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:27.684363 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:40:27.684371 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:40:27.684385 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:40:27.684437 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:40:27.687077 (+ 2640us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.23.100.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 10:40:27.687329 (+ 252us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44680: Request ticket server kudu/127.23.100.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":296,"threads_started":1}
W20260925 10:40:27.687654 23952 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 463: Not authorized: Client connection negotiation failed: client connection to 127.23.100.2:39099: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.23.100.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260925 10:40:27.687709 775 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44680: Request ticket server kudu/127.23.100.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 10:40:27.687786 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 463
I20260925 10:40:27.703961 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 596
I20260925 10:40:27.715929 776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:27.704962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33898 (local address 127.23.100.3:45801)
0925 10:40:27.705623 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:27.705632 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:27.705663 (+ 31us) server_negotiation.cc:407] Connection header received
0925 10:40:27.705745 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:27.705754 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:27.706043 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:27.706255 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:27.707501 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:27.708744 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:27.709884 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:27.710546 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:27.710646 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:27.712934 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:40:27.712942 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:40:27.712955 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:40:27.712991 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:40:27.715377 (+ 2386us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.23.100.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 10:40:27.715636 (+ 259us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33898: Request ticket server kudu/127.23.100.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":310,"threads_started":1}
W20260925 10:40:27.715976 23952 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 596: Not authorized: Client connection negotiation failed: client connection to 127.23.100.3:45801: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.23.100.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260925 10:40:27.716037 776 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33898: Request ticket server kudu/127.23.100.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 10:40:27.716284 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 596
I20260925 10:40:27.732422 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 32711
I20260925 10:40:27.733660 305 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:40:27.850875 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 32711
2026-09-25T10:40:27Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20955 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T10:40:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:40:27Z Disabled control of system clock
I20260925 10:40:27.923666 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:42361
--webserver_interface=127.23.100.62
--webserver_port=0
--builtin_ntp_servers=127.23.100.20:39425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.62:42361
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 10:40:28.233930 781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:40:28.234205 781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:40:28.234280 781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:40:28.241338 781 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:40:28.241439 781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:40:28.241487 781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:40:28.241542 781 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:40:28.241585 781 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:40:28.250346 781 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:39425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.62:42361
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:42361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:40:28.251520 781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:40:28.253156 781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:40:28.264724 787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:40:28.264802 786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:40:28.264793 789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:40:28.265287 781 server_base.cc:1034] running on GCE node
I20260925 10:40:28.266075 781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:40:28.267391 781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:40:28.268590 781 hybrid_clock.cc:648] HybridClock initialized: now 1790332828268542 us; error 42 us; skew 500 ppm
I20260925 10:40:28.268946 781 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:40:28.271232 781 webserver.cc:460] Webserver started at http://127.23.100.62:40881/ using document root <none> and password file <none>
I20260925 10:40:28.271898 781 fs_manager.cc:359] Metadata directory not provided
I20260925 10:40:28.272014 781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:40:28.272375 781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:40:28.275233 781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/master-0/data/instance:
uuid: "fd803e87409342689f6c0a9e5821877a"
format_stamp: "Formatted at 2026-09-25 10:40:28 on dist-test-slave-dxdd"
server_key: "b379c03535f97068a57fbc3474ad05f4"
server_key_iv: "19cf9203109a91ee5f21d651852edfe2"
server_key_version: "encryptionkey@0"
I20260925 10:40:28.276014 781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/master-0/wal/instance:
uuid: "fd803e87409342689f6c0a9e5821877a"
format_stamp: "Formatted at 2026-09-25 10:40:28 on dist-test-slave-dxdd"
server_key: "b379c03535f97068a57fbc3474ad05f4"
server_key_iv: "19cf9203109a91ee5f21d651852edfe2"
server_key_version: "encryptionkey@0"
I20260925 10:40:28.280789 781 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260925 10:40:28.284016 795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:40:28.285250 781 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 10:40:28.285429 781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/master-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/master-0/wal
uuid: "fd803e87409342689f6c0a9e5821877a"
format_stamp: "Formatted at 2026-09-25 10:40:28 on dist-test-slave-dxdd"
server_key: "b379c03535f97068a57fbc3474ad05f4"
server_key_iv: "19cf9203109a91ee5f21d651852edfe2"
server_key_version: "encryptionkey@0"
I20260925 10:40:28.285642 781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:40:28.313838 781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:40:28.314709 781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:40:28.315003 781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:40:28.332772 781 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.62:42361
I20260925 10:40:28.332792 846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.62:42361 every 8 connection(s)
I20260925 10:40:28.334021 781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
I20260925 10:40:28.337598 847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:40:28.341128 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 781
I20260925 10:40:28.341298 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/master-0/wal/instance
I20260925 10:40:28.341930 23952 external_mini_cluster.cc:1428] Setting key 9953ea1f1fd35a428f55961e5e872fde
I20260925 10:40:28.348143 847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd803e87409342689f6c0a9e5821877a: Bootstrap starting.
I20260925 10:40:28.351392 847 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd803e87409342689f6c0a9e5821877a: Neither blocks nor log segments found. Creating new log.
I20260925 10:40:28.352548 847 log.cc:824] T 00000000000000000000000000000000 P fd803e87409342689f6c0a9e5821877a: Log is configured to *not* fsync() on all Append() calls
I20260925 10:40:28.355216 847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd803e87409342689f6c0a9e5821877a: No bootstrap required, opened a new log
I20260925 10:40:28.355306 850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:28.343944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36122 (local address 127.23.100.62:42361)
0925 10:40:28.344881 (+ 937us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:28.344892 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:28.344934 (+ 42us) server_negotiation.cc:407] Connection header received
0925 10:40:28.346758 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:28.346777 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:28.347169 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:28.347465 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:28.348139 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:28.349348 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:28.351174 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:28.351866 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:28.351993 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:28.353181 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:40:28.353187 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:40:28.353199 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:40:28.353702 (+ 503us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:40:28.353768 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:40:28.353788 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:40:28.353919 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:28.354119 (+ 200us) server_negotiation.cc:299] Negotiation successful
0925 10:40:28.354459 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":324,"threads_started":1}
I20260925 10:40:28.362661 847 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd803e87409342689f6c0a9e5821877a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd803e87409342689f6c0a9e5821877a" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 42361 } }
I20260925 10:40:28.363061 847 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd803e87409342689f6c0a9e5821877a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:40:28.363178 847 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd803e87409342689f6c0a9e5821877a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd803e87409342689f6c0a9e5821877a, State: Initialized, Role: FOLLOWER
I20260925 10:40:28.363718 847 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd803e87409342689f6c0a9e5821877a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd803e87409342689f6c0a9e5821877a" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 42361 } }
I20260925 10:40:28.363960 847 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd803e87409342689f6c0a9e5821877a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:40:28.364101 847 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd803e87409342689f6c0a9e5821877a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:40:28.364274 847 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd803e87409342689f6c0a9e5821877a [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:40:28.366413 847 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd803e87409342689f6c0a9e5821877a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd803e87409342689f6c0a9e5821877a" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 42361 } }
I20260925 10:40:28.366933 847 leader_election.cc:302] T 00000000000000000000000000000000 P fd803e87409342689f6c0a9e5821877a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd803e87409342689f6c0a9e5821877a; no voters:
I20260925 10:40:28.367610 847 leader_election.cc:288] T 00000000000000000000000000000000 P fd803e87409342689f6c0a9e5821877a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:40:28.367753 852 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd803e87409342689f6c0a9e5821877a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:40:28.368530 852 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd803e87409342689f6c0a9e5821877a [term 1 LEADER]: Becoming Leader. State: Replica: fd803e87409342689f6c0a9e5821877a, State: Running, Role: LEADER
I20260925 10:40:28.369213 852 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd803e87409342689f6c0a9e5821877a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd803e87409342689f6c0a9e5821877a" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 42361 } }
I20260925 10:40:28.370600 847 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd803e87409342689f6c0a9e5821877a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:40:28.372539 853 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd803e87409342689f6c0a9e5821877a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd803e87409342689f6c0a9e5821877a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd803e87409342689f6c0a9e5821877a" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 42361 } } }
I20260925 10:40:28.372537 854 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd803e87409342689f6c0a9e5821877a [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd803e87409342689f6c0a9e5821877a. Latest consensus state: current_term: 1 leader_uuid: "fd803e87409342689f6c0a9e5821877a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd803e87409342689f6c0a9e5821877a" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 42361 } } }
I20260925 10:40:28.373006 853 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd803e87409342689f6c0a9e5821877a [sys.catalog]: This master's current role is: LEADER
I20260925 10:40:28.373090 854 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd803e87409342689f6c0a9e5821877a [sys.catalog]: This master's current role is: LEADER
I20260925 10:40:28.374895 857 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:40:28.382392 857 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:40:28.393743 857 catalog_manager.cc:1347] Generated new cluster ID: 3697c209c1b34feab1ff7fb884e9b12d
I20260925 10:40:28.393875 857 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:40:28.425209 857 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:40:28.426632 857 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:40:28.433308 857 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd803e87409342689f6c0a9e5821877a: Generated new TSK 0
I20260925 10:40:28.434244 857 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:40:28.440989 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.1:0
--local_ip_for_outbound_sockets=127.23.100.1
--webserver_interface=127.23.100.1
--webserver_port=0
--tserver_master_addrs=127.23.100.62:42361
--builtin_ntp_servers=127.23.100.20:39425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 10:40:28.755091 871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:40:28.755378 871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:40:28.755537 871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:40:28.762574 871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:40:28.762661 871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:40:28.762861 871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.1
I20260925 10:40:28.771668 871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:39425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.1
--webserver_port=0
--tserver_master_addrs=127.23.100.62:42361
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.1
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:40:28.772945 871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:40:28.774731 871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:40:28.787740 879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:40:28.787791 877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:40:28.787825 876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:40:28.788738 871 server_base.cc:1034] running on GCE node
I20260925 10:40:28.789343 871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:40:28.790360 871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:40:28.791613 871 hybrid_clock.cc:648] HybridClock initialized: now 1790332828791564 us; error 43 us; skew 500 ppm
I20260925 10:40:28.791983 871 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:40:28.797571 871 webserver.cc:460] Webserver started at http://127.23.100.1:34405/ using document root <none> and password file <none>
I20260925 10:40:28.798264 871 fs_manager.cc:359] Metadata directory not provided
I20260925 10:40:28.798456 871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:40:28.798775 871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:40:28.801578 871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-0/data/instance:
uuid: "75e0b24f3bcf4a9ca36487628a4cc45e"
format_stamp: "Formatted at 2026-09-25 10:40:28 on dist-test-slave-dxdd"
server_key: "6ce57cde1b1421dc5f5426639fe2df01"
server_key_iv: "3f139f4dc131fe971145e7ed532a1c7c"
server_key_version: "encryptionkey@0"
I20260925 10:40:28.802507 871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-0/wal/instance:
uuid: "75e0b24f3bcf4a9ca36487628a4cc45e"
format_stamp: "Formatted at 2026-09-25 10:40:28 on dist-test-slave-dxdd"
server_key: "6ce57cde1b1421dc5f5426639fe2df01"
server_key_iv: "3f139f4dc131fe971145e7ed532a1c7c"
server_key_version: "encryptionkey@0"
I20260925 10:40:28.807519 871 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 10:40:28.810817 885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:40:28.812093 871 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 10:40:28.812269 871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-0/wal
uuid: "75e0b24f3bcf4a9ca36487628a4cc45e"
format_stamp: "Formatted at 2026-09-25 10:40:28 on dist-test-slave-dxdd"
server_key: "6ce57cde1b1421dc5f5426639fe2df01"
server_key_iv: "3f139f4dc131fe971145e7ed532a1c7c"
server_key_version: "encryptionkey@0"
I20260925 10:40:28.812495 871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:40:28.829165 871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:40:28.829933 871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:40:28.830255 871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:40:28.831190 871 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:40:28.832463 871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:40:28.832580 871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:40:28.832723 871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:40:28.832806 871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:40:28.864790 871 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.1:39573
I20260925 10:40:28.864837 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.1:39573 every 8 connection(s)
I20260925 10:40:28.866044 871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
I20260925 10:40:28.873737 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 871
I20260925 10:40:28.873944 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-0/wal/instance
I20260925 10:40:28.874811 23952 external_mini_cluster.cc:1428] Setting key 46cf56f4313e0bf6757e0c49b5c8f52b
I20260925 10:40:28.877776 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.2:0
--local_ip_for_outbound_sockets=127.23.100.2
--webserver_interface=127.23.100.2
--webserver_port=0
--tserver_master_addrs=127.23.100.62:42361
--builtin_ntp_servers=127.23.100.20:39425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 10:40:28.878793 1000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:28.868538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:32915 (local address 127.23.100.62:42361)
0925 10:40:28.868955 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:28.868959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:28.870080 (+ 1121us) server_negotiation.cc:407] Connection header received
0925 10:40:28.871829 (+ 1749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:28.871833 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:28.871915 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:28.872002 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:28.872967 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:28.873717 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:28.875947 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:28.876539 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:28.876673 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:28.878106 (+ 1433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:40:28.878110 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:40:28.878113 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:40:28.878128 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:40:28.878165 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:40:28.878176 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:40:28.878239 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:28.878507 (+ 268us) server_negotiation.cc:299] Negotiation successful
0925 10:40:28.878573 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":229,"threads_started":1}
I20260925 10:40:28.880188 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:28.869131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:42361 (local address 127.23.100.1:32915)
0925 10:40:28.869938 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:28.869971 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:28.871562 (+ 1591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:28.872201 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:28.872208 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:28.872286 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:28.872807 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:28.872823 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:28.873885 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:28.873891 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:28.875818 (+ 1927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:28.875830 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:28.876794 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:28.876801 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:28.877084 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:28.877767 (+ 683us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:40:28.877772 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:28.877822 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:40:28.877925 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:40:28.878270 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:40:28.878282 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:40:28.878293 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 10:40:28.878487 (+ 194us) client_negotiation.cc:240] Negotiation successful
0925 10:40:28.878865 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":253,"threads_started":1}
I20260925 10:40:28.881726 998 heartbeater.cc:347] Connected to a master server at 127.23.100.62:42361
I20260925 10:40:28.882014 998 heartbeater.cc:464] Registering TS with master...
I20260925 10:40:28.882880 998 heartbeater.cc:511] Master 127.23.100.62:42361 requested a full tablet report, sending...
I20260925 10:40:28.885113 812 ts_manager.cc:194] Registered new tserver with Master: 75e0b24f3bcf4a9ca36487628a4cc45e (127.23.100.1:39573)
I20260925 10:40:28.886701 812 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.100.1:32915
W20260925 10:40:29.190950 1002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:40:29.191262 1002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:40:29.191421 1002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:40:29.198055 1002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:40:29.198189 1002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:40:29.198421 1002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.2
I20260925 10:40:29.206688 1002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:39425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.2
--webserver_port=0
--tserver_master_addrs=127.23.100.62:42361
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.2
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:40:29.207828 1002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:40:29.209468 1002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:40:29.221957 1008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:40:29.221918 1010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:40:29.221997 1007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:40:29.222728 1002 server_base.cc:1034] running on GCE node
I20260925 10:40:29.223361 1002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:40:29.224224 1002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:40:29.225438 1002 hybrid_clock.cc:648] HybridClock initialized: now 1790332829225395 us; error 59 us; skew 500 ppm
I20260925 10:40:29.225735 1002 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:40:29.227995 1002 webserver.cc:460] Webserver started at http://127.23.100.2:34523/ using document root <none> and password file <none>
I20260925 10:40:29.228637 1002 fs_manager.cc:359] Metadata directory not provided
I20260925 10:40:29.228765 1002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:40:29.229117 1002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:40:29.231845 1002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-1/data/instance:
uuid: "1c160f255b1944448d900bf12cfd2299"
format_stamp: "Formatted at 2026-09-25 10:40:29 on dist-test-slave-dxdd"
server_key: "def1dfa5b1864ee7128bd8286f664b05"
server_key_iv: "651ef68f4c298c23fab44b36b171d1b9"
server_key_version: "encryptionkey@0"
I20260925 10:40:29.232615 1002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-1/wal/instance:
uuid: "1c160f255b1944448d900bf12cfd2299"
format_stamp: "Formatted at 2026-09-25 10:40:29 on dist-test-slave-dxdd"
server_key: "def1dfa5b1864ee7128bd8286f664b05"
server_key_iv: "651ef68f4c298c23fab44b36b171d1b9"
server_key_version: "encryptionkey@0"
I20260925 10:40:29.237183 1002 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260925 10:40:29.240322 1016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:40:29.241516 1002 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 10:40:29.241677 1002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-1/wal
uuid: "1c160f255b1944448d900bf12cfd2299"
format_stamp: "Formatted at 2026-09-25 10:40:29 on dist-test-slave-dxdd"
server_key: "def1dfa5b1864ee7128bd8286f664b05"
server_key_iv: "651ef68f4c298c23fab44b36b171d1b9"
server_key_version: "encryptionkey@0"
I20260925 10:40:29.241875 1002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:40:29.259064 1002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:40:29.259783 1002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:40:29.260108 1002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:40:29.260881 1002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:40:29.262089 1002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:40:29.262168 1002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:40:29.262243 1002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:40:29.262295 1002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:40:29.293797 1002 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.2:40245
I20260925 10:40:29.293901 1128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.2:40245 every 8 connection(s)
I20260925 10:40:29.295109 1002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
I20260925 10:40:29.300874 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 1002
I20260925 10:40:29.301043 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-1/wal/instance
I20260925 10:40:29.301596 23952 external_mini_cluster.cc:1428] Setting key f4dbf58f9bac64cd38a1f202454c612f
I20260925 10:40:29.304234 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.3:0
--local_ip_for_outbound_sockets=127.23.100.3
--webserver_interface=127.23.100.3
--webserver_port=0
--tserver_master_addrs=127.23.100.62:42361
--builtin_ntp_servers=127.23.100.20:39425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 10:40:29.307878 1000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:29.297437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:40285 (local address 127.23.100.62:42361)
0925 10:40:29.297648 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:29.297653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:29.299049 (+ 1396us) server_negotiation.cc:407] Connection header received
0925 10:40:29.300729 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:29.300733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:29.300796 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:29.300877 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:29.302240 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:29.303144 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:29.304888 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:29.305647 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:29.305790 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:29.306992 (+ 1202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:40:29.307001 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:40:29.307008 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:40:29.307029 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:40:29.307069 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:40:29.307085 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:40:29.307193 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:29.307378 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 10:40:29.307489 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 10:40:29.308621 1131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:29.297953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:42361 (local address 127.23.100.2:40285)
0925 10:40:29.298890 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:29.298940 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:29.300429 (+ 1489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:29.301150 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:29.301162 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:29.301268 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:29.302015 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:29.302039 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:29.303343 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:29.303348 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:29.304729 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:29.304745 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:29.305837 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:29.305841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:29.306026 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:29.306715 (+ 689us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:40:29.306717 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:29.306750 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:40:29.306815 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:40:29.307195 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:40:29.307202 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:40:29.307207 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:40:29.307336 (+ 129us) client_negotiation.cc:240] Negotiation successful
0925 10:40:29.307613 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":311,"threads_started":1}
I20260925 10:40:29.309880 1129 heartbeater.cc:347] Connected to a master server at 127.23.100.62:42361
I20260925 10:40:29.310186 1129 heartbeater.cc:464] Registering TS with master...
I20260925 10:40:29.311046 1129 heartbeater.cc:511] Master 127.23.100.62:42361 requested a full tablet report, sending...
I20260925 10:40:29.312525 812 ts_manager.cc:194] Registered new tserver with Master: 1c160f255b1944448d900bf12cfd2299 (127.23.100.2:40245)
I20260925 10:40:29.313316 812 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.100.2:40285
W20260925 10:40:29.616037 1132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:40:29.616277 1132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:40:29.616441 1132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:40:29.623580 1132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:40:29.623735 1132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:40:29.623919 1132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.3
I20260925 10:40:29.632733 1132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:39425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.3
--webserver_port=0
--tserver_master_addrs=127.23.100.62:42361
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.3
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:40:29.633924 1132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:40:29.635737 1132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:40:29.648751 1140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:40:29.648829 1138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:40:29.649183 1137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:40:29.649364 1132 server_base.cc:1034] running on GCE node
I20260925 10:40:29.649963 1132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:40:29.650900 1132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:40:29.652122 1132 hybrid_clock.cc:648] HybridClock initialized: now 1790332829652082 us; error 56 us; skew 500 ppm
I20260925 10:40:29.652431 1132 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:40:29.654563 1132 webserver.cc:460] Webserver started at http://127.23.100.3:37915/ using document root <none> and password file <none>
I20260925 10:40:29.655239 1132 fs_manager.cc:359] Metadata directory not provided
I20260925 10:40:29.655402 1132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:40:29.655717 1132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:40:29.658270 1132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-2/data/instance:
uuid: "205a509d44e74035a6cf923054714e64"
format_stamp: "Formatted at 2026-09-25 10:40:29 on dist-test-slave-dxdd"
server_key: "9ee36b66d2bbde5ae4dcb91055a88a98"
server_key_iv: "46bf6475d1c482dc9293eefed43c36bc"
server_key_version: "encryptionkey@0"
I20260925 10:40:29.659021 1132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-2/wal/instance:
uuid: "205a509d44e74035a6cf923054714e64"
format_stamp: "Formatted at 2026-09-25 10:40:29 on dist-test-slave-dxdd"
server_key: "9ee36b66d2bbde5ae4dcb91055a88a98"
server_key_iv: "46bf6475d1c482dc9293eefed43c36bc"
server_key_version: "encryptionkey@0"
I20260925 10:40:29.663718 1132 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 10:40:29.666891 1146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:40:29.668054 1132 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 10:40:29.668213 1132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-2/wal
uuid: "205a509d44e74035a6cf923054714e64"
format_stamp: "Formatted at 2026-09-25 10:40:29 on dist-test-slave-dxdd"
server_key: "9ee36b66d2bbde5ae4dcb91055a88a98"
server_key_iv: "46bf6475d1c482dc9293eefed43c36bc"
server_key_version: "encryptionkey@0"
I20260925 10:40:29.668408 1132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:40:29.683236 1132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:40:29.683991 1132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:40:29.684271 1132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:40:29.685122 1132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:40:29.686504 1132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:40:29.686630 1132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:40:29.686759 1132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:40:29.686846 1132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:40:29.717757 1132 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.3:46383
I20260925 10:40:29.717803 1258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.3:46383 every 8 connection(s)
I20260925 10:40:29.719058 1132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
I20260925 10:40:29.726591 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 1132
I20260925 10:40:29.726738 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-2/wal/instance
I20260925 10:40:29.727171 23952 external_mini_cluster.cc:1428] Setting key b4c9414cf891f470cef6933a7f82a0b2
I20260925 10:40:29.731081 1000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:29.721555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:54727 (local address 127.23.100.62:42361)
0925 10:40:29.721761 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:29.721767 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:29.723122 (+ 1355us) server_negotiation.cc:407] Connection header received
0925 10:40:29.724675 (+ 1553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:29.724680 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:29.724742 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:29.724824 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:29.725761 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:29.726731 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:29.728430 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:29.728938 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:29.729087 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:29.730462 (+ 1375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:40:29.730466 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:40:29.730470 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:40:29.730485 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:40:29.730513 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:40:29.730525 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:40:29.730591 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:29.730769 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 10:40:29.730836 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 10:40:29.731976 1261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:29.722110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:42361 (local address 127.23.100.3:54727)
0925 10:40:29.722981 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:29.723015 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:29.724459 (+ 1444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:29.725013 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:29.725020 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:29.725098 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:29.725627 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:29.725641 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:29.726896 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:29.726900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:29.728271 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:29.728285 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:29.729149 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:29.729155 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:29.729341 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:29.730090 (+ 749us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:40:29.730093 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:29.730146 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:40:29.730258 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:40:29.730602 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:40:29.730609 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:40:29.730615 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 10:40:29.730743 (+ 128us) client_negotiation.cc:240] Negotiation successful
0925 10:40:29.730995 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":312,"threads_started":1}
I20260925 10:40:29.733119 1259 heartbeater.cc:347] Connected to a master server at 127.23.100.62:42361
I20260925 10:40:29.733379 1259 heartbeater.cc:464] Registering TS with master...
I20260925 10:40:29.734037 1259 heartbeater.cc:511] Master 127.23.100.62:42361 requested a full tablet report, sending...
I20260925 10:40:29.735265 812 ts_manager.cc:194] Registered new tserver with Master: 205a509d44e74035a6cf923054714e64 (127.23.100.3:46383)
I20260925 10:40:29.736027 812 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.100.3:54727
I20260925 10:40:29.741863 23952 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 10:40:29.747558 1000 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:40:29.746138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36128 (local address 127.23.100.62:42361)
0925 10:40:29.746341 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:29.746349 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:29.746863 (+ 514us) server_negotiation.cc:407] Connection header received
0925 10:40:29.746979 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:29.746983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:29.747047 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:29.747118 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:29.747406 (+ 288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36128: BlockingRecv error: recv got EOF from 127.0.0.1:36128 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 10:40:29.748359 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 871
I20260925 10:40:29.755396 1268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:29.749233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59782 (local address 127.23.100.1:39573)
0925 10:40:29.749722 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:29.749729 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:29.749753 (+ 24us) server_negotiation.cc:407] Connection header received
0925 10:40:29.749834 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:29.749839 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:29.750009 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:29.750139 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:29.750512 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:29.751458 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:29.752602 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:29.753086 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:29.753174 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:29.754101 (+ 927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:40:29.754108 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:40:29.754118 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:40:29.754722 (+ 604us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:40:29.754769 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:40:29.754787 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:40:29.754869 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:29.755002 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 10:40:29.755115 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":259,"threads_started":1}
I20260925 10:40:29.756516 993 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:40:29.837545 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 871
I20260925 10:40:29.854796 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 1002
I20260925 10:40:29.862702 1270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:29.855732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56032 (local address 127.23.100.2:40245)
0925 10:40:29.856243 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:29.856249 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:29.856276 (+ 27us) server_negotiation.cc:407] Connection header received
0925 10:40:29.856333 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:29.856339 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:29.856558 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:29.856683 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:29.857278 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:29.858262 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:29.859629 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:29.860205 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:29.860300 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:29.861191 (+ 891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:40:29.861199 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:40:29.861212 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:40:29.861769 (+ 557us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:40:29.861812 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:40:29.861829 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:40:29.861920 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:29.862074 (+ 154us) server_negotiation.cc:299] Negotiation successful
0925 10:40:29.862230 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":270,"threads_started":1}
I20260925 10:40:29.863903 1124 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:40:29.943315 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 1002
I20260925 10:40:29.960228 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 1132
I20260925 10:40:29.967928 1272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:29.961264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42090 (local address 127.23.100.3:46383)
0925 10:40:29.961920 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:29.961926 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:29.961956 (+ 30us) server_negotiation.cc:407] Connection header received
0925 10:40:29.962030 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:29.962036 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:29.962243 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:29.962434 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:29.962775 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:29.963782 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:29.964940 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:29.965474 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:29.965566 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:29.966535 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:40:29.966544 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:40:29.966556 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:40:29.967121 (+ 565us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:40:29.967169 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:40:29.967188 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:40:29.967279 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:29.967413 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 10:40:29.967571 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":360,"threads_started":1}
I20260925 10:40:29.969107 1254 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:40:30.049767 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 1132
I20260925 10:40:30.065233 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 781
I20260925 10:40:30.066422 842 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:40:30.166620 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 781
2026-09-25T10:40:30Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2310 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T10:40:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:40:30Z Disabled control of system clock
I20260925 10:40:30.233667 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:38515
--webserver_interface=127.23.100.62
--webserver_port=0
--builtin_ntp_servers=127.23.100.20:35195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.62:38515
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 10:40:30.540210 1278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:40:30.540459 1278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:40:30.540632 1278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:40:30.547514 1278 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:40:30.547595 1278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:40:30.547632 1278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:40:30.547679 1278 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:40:30.547763 1278 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:40:30.556483 1278 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:35195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.62:38515
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:38515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:40:30.557621 1278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:40:30.559351 1278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:40:30.570909 1283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:40:30.570824 1286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:40:30.570869 1284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:40:30.571600 1278 server_base.cc:1034] running on GCE node
I20260925 10:40:30.572301 1278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:40:30.573549 1278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:40:30.574743 1278 hybrid_clock.cc:648] HybridClock initialized: now 1790332830574710 us; error 29 us; skew 500 ppm
I20260925 10:40:30.575078 1278 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:40:30.577127 1278 webserver.cc:460] Webserver started at http://127.23.100.62:43235/ using document root <none> and password file <none>
I20260925 10:40:30.577778 1278 fs_manager.cc:359] Metadata directory not provided
I20260925 10:40:30.577913 1278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:40:30.578223 1278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:40:30.580914 1278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/master-0/data/instance:
uuid: "e0e067ceb8824786953573b0b0256856"
format_stamp: "Formatted at 2026-09-25 10:40:30 on dist-test-slave-dxdd"
server_key: "0c521e7ebd20594a4e8b0287160acbb2"
server_key_iv: "6a13d25ecd64121a7e91f9f59471e192"
server_key_version: "encryptionkey@0"
I20260925 10:40:30.581724 1278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/master-0/wal/instance:
uuid: "e0e067ceb8824786953573b0b0256856"
format_stamp: "Formatted at 2026-09-25 10:40:30 on dist-test-slave-dxdd"
server_key: "0c521e7ebd20594a4e8b0287160acbb2"
server_key_iv: "6a13d25ecd64121a7e91f9f59471e192"
server_key_version: "encryptionkey@0"
I20260925 10:40:30.586616 1278 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 10:40:30.589792 1292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:40:30.591068 1278 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:40:30.591239 1278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/master-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/master-0/wal
uuid: "e0e067ceb8824786953573b0b0256856"
format_stamp: "Formatted at 2026-09-25 10:40:30 on dist-test-slave-dxdd"
server_key: "0c521e7ebd20594a4e8b0287160acbb2"
server_key_iv: "6a13d25ecd64121a7e91f9f59471e192"
server_key_version: "encryptionkey@0"
I20260925 10:40:30.591445 1278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:40:30.596873 1278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:40:30.597575 1278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:40:30.597855 1278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:40:30.615409 1278 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.62:38515
I20260925 10:40:30.615445 1343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.62:38515 every 8 connection(s)
I20260925 10:40:30.616724 1278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
I20260925 10:40:30.620110 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 1278
I20260925 10:40:30.620328 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/master-0/wal/instance
I20260925 10:40:30.620618 1344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:40:30.621143 23952 external_mini_cluster.cc:1428] Setting key 26783454970a736064a128ad3c20e198
I20260925 10:40:30.629122 1347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:30.623073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45320 (local address 127.23.100.62:38515)
0925 10:40:30.624083 (+ 1010us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:30.624093 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:30.624129 (+ 36us) server_negotiation.cc:407] Connection header received
0925 10:40:30.625687 (+ 1558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:30.625711 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:30.626123 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:30.626578 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:30.627627 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:40:30.627633 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:40:30.627648 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:40:30.627665 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:40:30.627716 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:40:30.627729 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:40:30.627969 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:30.628108 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 10:40:30.628339 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":399,"threads_started":1}
I20260925 10:40:30.633273 1344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0e067ceb8824786953573b0b0256856: Bootstrap starting.
I20260925 10:40:30.636029 1344 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0e067ceb8824786953573b0b0256856: Neither blocks nor log segments found. Creating new log.
I20260925 10:40:30.637188 1344 log.cc:824] T 00000000000000000000000000000000 P e0e067ceb8824786953573b0b0256856: Log is configured to *not* fsync() on all Append() calls
I20260925 10:40:30.639684 1344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0e067ceb8824786953573b0b0256856: No bootstrap required, opened a new log
I20260925 10:40:30.647037 1344 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0e067ceb8824786953573b0b0256856 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0e067ceb8824786953573b0b0256856" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 38515 } }
I20260925 10:40:30.647359 1344 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0e067ceb8824786953573b0b0256856 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:40:30.647428 1344 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0e067ceb8824786953573b0b0256856 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0e067ceb8824786953573b0b0256856, State: Initialized, Role: FOLLOWER
I20260925 10:40:30.647984 1344 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0e067ceb8824786953573b0b0256856 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0e067ceb8824786953573b0b0256856" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 38515 } }
I20260925 10:40:30.648192 1344 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0e067ceb8824786953573b0b0256856 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:40:30.648284 1344 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0e067ceb8824786953573b0b0256856 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:40:30.648398 1344 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0e067ceb8824786953573b0b0256856 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:40:30.650697 1344 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0e067ceb8824786953573b0b0256856 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0e067ceb8824786953573b0b0256856" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 38515 } }
I20260925 10:40:30.651177 1344 leader_election.cc:302] T 00000000000000000000000000000000 P e0e067ceb8824786953573b0b0256856 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0e067ceb8824786953573b0b0256856; no voters:
I20260925 10:40:30.651736 1344 leader_election.cc:288] T 00000000000000000000000000000000 P e0e067ceb8824786953573b0b0256856 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:40:30.651859 1349 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0e067ceb8824786953573b0b0256856 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:40:30.652552 1349 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0e067ceb8824786953573b0b0256856 [term 1 LEADER]: Becoming Leader. State: Replica: e0e067ceb8824786953573b0b0256856, State: Running, Role: LEADER
I20260925 10:40:30.653134 1349 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0e067ceb8824786953573b0b0256856 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0e067ceb8824786953573b0b0256856" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 38515 } }
I20260925 10:40:30.653769 1344 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0e067ceb8824786953573b0b0256856 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:40:30.656116 1351 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0e067ceb8824786953573b0b0256856 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0e067ceb8824786953573b0b0256856. Latest consensus state: current_term: 1 leader_uuid: "e0e067ceb8824786953573b0b0256856" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0e067ceb8824786953573b0b0256856" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 38515 } } }
I20260925 10:40:30.656579 1351 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0e067ceb8824786953573b0b0256856 [sys.catalog]: This master's current role is: LEADER
I20260925 10:40:30.656186 1350 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0e067ceb8824786953573b0b0256856 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0e067ceb8824786953573b0b0256856" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0e067ceb8824786953573b0b0256856" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 38515 } } }
I20260925 10:40:30.657202 1350 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0e067ceb8824786953573b0b0256856 [sys.catalog]: This master's current role is: LEADER
I20260925 10:40:30.661123 1358 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:40:30.666627 1358 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:40:30.674228 1358 catalog_manager.cc:1347] Generated new cluster ID: f3e8adf765854a17846fb040af3338f5
I20260925 10:40:30.674389 1358 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:40:30.689531 1358 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:40:30.689936 1358 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:40:30.701653 1358 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0e067ceb8824786953573b0b0256856: Generated new TSK 0
I20260925 10:40:30.702554 1358 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:40:30.712481 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.1:0
--local_ip_for_outbound_sockets=127.23.100.1
--webserver_interface=127.23.100.1
--webserver_port=0
--tserver_master_addrs=127.23.100.62:38515
--builtin_ntp_servers=127.23.100.20:35195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 10:40:31.034664 1368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:40:31.034941 1368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:40:31.035032 1368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:40:31.042128 1368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:40:31.042215 1368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:40:31.042382 1368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.1
I20260925 10:40:31.055567 1368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:35195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.1
--webserver_port=0
--tserver_master_addrs=127.23.100.62:38515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.1
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:40:31.056829 1368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:40:31.058601 1368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:40:31.074750 1374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:40:31.074913 1373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:40:31.074863 1376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:40:31.075513 1368 server_base.cc:1034] running on GCE node
I20260925 10:40:31.076051 1368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:40:31.076877 1368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:40:31.078068 1368 hybrid_clock.cc:648] HybridClock initialized: now 1790332831078021 us; error 37 us; skew 500 ppm
I20260925 10:40:31.078454 1368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:40:31.080617 1368 webserver.cc:460] Webserver started at http://127.23.100.1:37679/ using document root <none> and password file <none>
I20260925 10:40:31.081286 1368 fs_manager.cc:359] Metadata directory not provided
I20260925 10:40:31.081393 1368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:40:31.081701 1368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:40:31.084446 1368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-0/data/instance:
uuid: "c304cd8c5cb048859232453db31689b7"
format_stamp: "Formatted at 2026-09-25 10:40:31 on dist-test-slave-dxdd"
server_key: "71f0c15469218b9e720906b2766c26cf"
server_key_iv: "d29ae822d0a6b2851445b1d08a15b464"
server_key_version: "encryptionkey@0"
I20260925 10:40:31.085191 1368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-0/wal/instance:
uuid: "c304cd8c5cb048859232453db31689b7"
format_stamp: "Formatted at 2026-09-25 10:40:31 on dist-test-slave-dxdd"
server_key: "71f0c15469218b9e720906b2766c26cf"
server_key_iv: "d29ae822d0a6b2851445b1d08a15b464"
server_key_version: "encryptionkey@0"
I20260925 10:40:31.090025 1368 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 10:40:31.093269 1382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:40:31.094604 1368 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:40:31.094760 1368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-0/wal
uuid: "c304cd8c5cb048859232453db31689b7"
format_stamp: "Formatted at 2026-09-25 10:40:31 on dist-test-slave-dxdd"
server_key: "71f0c15469218b9e720906b2766c26cf"
server_key_iv: "d29ae822d0a6b2851445b1d08a15b464"
server_key_version: "encryptionkey@0"
I20260925 10:40:31.094980 1368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:40:31.100332 1368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:40:31.101126 1368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:40:31.101415 1368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:40:31.102264 1368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:40:31.103603 1368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:40:31.103724 1368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:40:31.103842 1368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:40:31.103924 1368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:40:31.135761 1368 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.1:38931
I20260925 10:40:31.135814 1494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.1:38931 every 8 connection(s)
I20260925 10:40:31.137235 1368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
I20260925 10:40:31.143357 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 1368
I20260925 10:40:31.143559 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-0/wal/instance
I20260925 10:40:31.144385 23952 external_mini_cluster.cc:1428] Setting key 5bdaeb7e430ba1b458232c985c460ce5
I20260925 10:40:31.145463 1497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:31.139746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:34919 (local address 127.23.100.62:38515)
0925 10:40:31.140145 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:31.140150 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:31.141358 (+ 1208us) server_negotiation.cc:407] Connection header received
0925 10:40:31.143078 (+ 1720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:31.143082 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:31.143135 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:31.143229 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:31.144693 (+ 1464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:40:31.144697 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:40:31.144701 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:40:31.144716 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:40:31.144756 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:40:31.144767 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:40:31.144862 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:31.145133 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 10:40:31.145198 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":196,"threads_started":1}
I20260925 10:40:31.146555 1498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:31.140353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:38515 (local address 127.23.100.1:34919)
0925 10:40:31.141195 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:31.141234 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:31.142779 (+ 1545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:31.143504 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:31.143517 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:31.143640 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:31.144339 (+ 699us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:40:31.144343 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:31.144395 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:40:31.144511 (+ 116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:40:31.144883 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:40:31.144893 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:40:31.144900 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 10:40:31.145146 (+ 246us) client_negotiation.cc:240] Negotiation successful
0925 10:40:31.145391 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":249,"threads_started":1}
I20260925 10:40:31.147392 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.2:0
--local_ip_for_outbound_sockets=127.23.100.2
--webserver_interface=127.23.100.2
--webserver_port=0
--tserver_master_addrs=127.23.100.62:38515
--builtin_ntp_servers=127.23.100.20:35195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 10:40:31.148033 1495 heartbeater.cc:347] Connected to a master server at 127.23.100.62:38515
I20260925 10:40:31.148313 1495 heartbeater.cc:464] Registering TS with master...
I20260925 10:40:31.148854 1495 heartbeater.cc:511] Master 127.23.100.62:38515 requested a full tablet report, sending...
I20260925 10:40:31.150882 1309 ts_manager.cc:194] Registered new tserver with Master: c304cd8c5cb048859232453db31689b7 (127.23.100.1:38931)
W20260925 10:40:31.460304 1499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:40:31.460602 1499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:40:31.460763 1499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:40:31.467892 1499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:40:31.467973 1499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:40:31.468163 1499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.2
I20260925 10:40:31.476596 1499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:35195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.2
--webserver_port=0
--tserver_master_addrs=127.23.100.62:38515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.2
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:40:31.477780 1499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:40:31.479524 1499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:40:31.492094 1507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:40:31.492132 1504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:40:31.492185 1505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:40:31.492858 1499 server_base.cc:1034] running on GCE node
I20260925 10:40:31.493408 1499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:40:31.494318 1499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:40:31.495540 1499 hybrid_clock.cc:648] HybridClock initialized: now 1790332831495500 us; error 51 us; skew 500 ppm
I20260925 10:40:31.495857 1499 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:40:31.497975 1499 webserver.cc:460] Webserver started at http://127.23.100.2:42411/ using document root <none> and password file <none>
I20260925 10:40:31.498692 1499 fs_manager.cc:359] Metadata directory not provided
I20260925 10:40:31.498817 1499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:40:31.499119 1499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:40:31.501772 1499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-1/data/instance:
uuid: "3a429c5674ce4d49a4c4bab748625678"
format_stamp: "Formatted at 2026-09-25 10:40:31 on dist-test-slave-dxdd"
server_key: "ba4c2bd9a44425bdb6c5104a9f9e0b04"
server_key_iv: "e5ae984460207589ce36b532500ab560"
server_key_version: "encryptionkey@0"
I20260925 10:40:31.502609 1499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-1/wal/instance:
uuid: "3a429c5674ce4d49a4c4bab748625678"
format_stamp: "Formatted at 2026-09-25 10:40:31 on dist-test-slave-dxdd"
server_key: "ba4c2bd9a44425bdb6c5104a9f9e0b04"
server_key_iv: "e5ae984460207589ce36b532500ab560"
server_key_version: "encryptionkey@0"
I20260925 10:40:31.507436 1499 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 10:40:31.510735 1513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:40:31.512007 1499 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 10:40:31.512207 1499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-1/wal
uuid: "3a429c5674ce4d49a4c4bab748625678"
format_stamp: "Formatted at 2026-09-25 10:40:31 on dist-test-slave-dxdd"
server_key: "ba4c2bd9a44425bdb6c5104a9f9e0b04"
server_key_iv: "e5ae984460207589ce36b532500ab560"
server_key_version: "encryptionkey@0"
I20260925 10:40:31.512436 1499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:40:31.517875 1499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:40:31.518724 1499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:40:31.518982 1499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:40:31.519865 1499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:40:31.521170 1499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:40:31.521294 1499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:40:31.521422 1499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:40:31.521507 1499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:40:31.552939 1499 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.2:43685
I20260925 10:40:31.553020 1625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.2:43685 every 8 connection(s)
I20260925 10:40:31.554258 1499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
I20260925 10:40:31.558843 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 1499
I20260925 10:40:31.559041 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-1/wal/instance
I20260925 10:40:31.559531 23952 external_mini_cluster.cc:1428] Setting key 906601f38e6e0f979cef3a60b5b4212e
I20260925 10:40:31.562435 1497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:31.556741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:55301 (local address 127.23.100.62:38515)
0925 10:40:31.556907 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:31.556912 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:31.558116 (+ 1204us) server_negotiation.cc:407] Connection header received
0925 10:40:31.560035 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:31.560041 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:31.560115 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:31.560228 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:31.561684 (+ 1456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:40:31.561706 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:40:31.561713 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:40:31.561733 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:40:31.561770 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:40:31.561783 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:40:31.561915 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:31.562120 (+ 205us) server_negotiation.cc:299] Negotiation successful
0925 10:40:31.562179 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 10:40:31.562674 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.3:0
--local_ip_for_outbound_sockets=127.23.100.3
--webserver_interface=127.23.100.3
--webserver_port=0
--tserver_master_addrs=127.23.100.62:38515
--builtin_ntp_servers=127.23.100.20:35195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 10:40:31.563562 1628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:31.557153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:38515 (local address 127.23.100.2:55301)
0925 10:40:31.557972 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:31.558005 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:31.559647 (+ 1642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:31.560464 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:31.560471 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:31.560602 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:31.561355 (+ 753us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:40:31.561357 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:31.561404 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:40:31.561500 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:40:31.561890 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:40:31.561900 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:40:31.561907 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 10:40:31.562173 (+ 266us) client_negotiation.cc:240] Negotiation successful
0925 10:40:31.562443 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":243,"threads_started":1}
I20260925 10:40:31.567267 1626 heartbeater.cc:347] Connected to a master server at 127.23.100.62:38515
I20260925 10:40:31.567535 1626 heartbeater.cc:464] Registering TS with master...
I20260925 10:40:31.568068 1626 heartbeater.cc:511] Master 127.23.100.62:38515 requested a full tablet report, sending...
I20260925 10:40:31.569527 1309 ts_manager.cc:194] Registered new tserver with Master: 3a429c5674ce4d49a4c4bab748625678 (127.23.100.2:43685)
W20260925 10:40:31.880338 1629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:40:31.880596 1629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:40:31.880683 1629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:40:31.887954 1629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:40:31.888044 1629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:40:31.888147 1629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.3
I20260925 10:40:31.896890 1629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:35195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.3
--webserver_port=0
--tserver_master_addrs=127.23.100.62:38515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.3
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:40:31.898100 1629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:40:31.899914 1629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:40:31.912504 1635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:40:31.912439 1637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:40:31.912822 1634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:40:31.913431 1629 server_base.cc:1034] running on GCE node
I20260925 10:40:31.913929 1629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:40:31.914834 1629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:40:31.916023 1629 hybrid_clock.cc:648] HybridClock initialized: now 1790332831915995 us; error 32 us; skew 500 ppm
I20260925 10:40:31.916316 1629 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:40:31.918587 1629 webserver.cc:460] Webserver started at http://127.23.100.3:41637/ using document root <none> and password file <none>
I20260925 10:40:31.919216 1629 fs_manager.cc:359] Metadata directory not provided
I20260925 10:40:31.919303 1629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:40:31.919586 1629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:40:31.922248 1629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-2/data/instance:
uuid: "091c8d59b20b43a1b9072d537476e673"
format_stamp: "Formatted at 2026-09-25 10:40:31 on dist-test-slave-dxdd"
server_key: "0b7e6a1d0bfa60daeb47fe57705c0fce"
server_key_iv: "b1acbbbf36e2a7f9641170ca1a0b578e"
server_key_version: "encryptionkey@0"
I20260925 10:40:31.923034 1629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-2/wal/instance:
uuid: "091c8d59b20b43a1b9072d537476e673"
format_stamp: "Formatted at 2026-09-25 10:40:31 on dist-test-slave-dxdd"
server_key: "0b7e6a1d0bfa60daeb47fe57705c0fce"
server_key_iv: "b1acbbbf36e2a7f9641170ca1a0b578e"
server_key_version: "encryptionkey@0"
I20260925 10:40:31.928316 1629 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260925 10:40:31.931442 1643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:40:31.932684 1629 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:40:31.932840 1629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-2/wal
uuid: "091c8d59b20b43a1b9072d537476e673"
format_stamp: "Formatted at 2026-09-25 10:40:31 on dist-test-slave-dxdd"
server_key: "0b7e6a1d0bfa60daeb47fe57705c0fce"
server_key_iv: "b1acbbbf36e2a7f9641170ca1a0b578e"
server_key_version: "encryptionkey@0"
I20260925 10:40:31.933017 1629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:40:31.938612 1629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:40:31.939288 1629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:40:31.939500 1629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:40:31.940336 1629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:40:31.941605 1629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:40:31.941686 1629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:40:31.941761 1629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:40:31.941821 1629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:40:31.974053 1629 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.3:33941
I20260925 10:40:31.974093 1755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.3:33941 every 8 connection(s)
I20260925 10:40:31.975368 1629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
I20260925 10:40:31.983052 1497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:31.977866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:55087 (local address 127.23.100.62:38515)
0925 10:40:31.978085 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:31.978091 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:31.979426 (+ 1335us) server_negotiation.cc:407] Connection header received
0925 10:40:31.981015 (+ 1589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:31.981019 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:31.981071 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:31.981158 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:31.982396 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:40:31.982400 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:40:31.982404 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:40:31.982421 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:40:31.982458 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:40:31.982469 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:40:31.982572 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:31.982756 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 10:40:31.982808 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 10:40:31.983872 1758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:31.978442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:38515 (local address 127.23.100.3:55087)
0925 10:40:31.979270 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:31.979306 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:31.980771 (+ 1465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:31.981336 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:31.981343 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:31.981426 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:31.982060 (+ 634us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:40:31.982062 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:31.982095 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:40:31.982168 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:40:31.982551 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:40:31.982559 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:40:31.982564 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:40:31.982797 (+ 233us) client_negotiation.cc:240] Negotiation successful
0925 10:40:31.982990 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":242,"threads_started":1}
I20260925 10:40:31.984355 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 1629
I20260925 10:40:31.984627 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790332740062128-23952-0/minicluster-data/ts-2/wal/instance
I20260925 10:40:31.985179 23952 external_mini_cluster.cc:1428] Setting key 2154403721d04af0c16dd47d5a7625e4
I20260925 10:40:31.985404 1756 heartbeater.cc:347] Connected to a master server at 127.23.100.62:38515
I20260925 10:40:31.985697 1756 heartbeater.cc:464] Registering TS with master...
I20260925 10:40:31.986362 1756 heartbeater.cc:511] Master 127.23.100.62:38515 requested a full tablet report, sending...
I20260925 10:40:31.988075 1308 ts_manager.cc:194] Registered new tserver with Master: 091c8d59b20b43a1b9072d537476e673 (127.23.100.3:33941)
I20260925 10:40:32.000576 23952 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 10:40:32.006385 1497 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:40:32.004896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49876 (local address 127.23.100.62:38515)
0925 10:40:32.005106 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:32.005114 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:32.005466 (+ 352us) server_negotiation.cc:407] Connection header received
0925 10:40:32.005628 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:32.005632 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:32.005686 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:32.005767 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:32.006078 (+ 311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49876: BlockingRecv error: recv got EOF from 127.0.0.1:49876 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 10:40:32.007217 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 1368
I20260925 10:40:32.010617 1765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:32.008124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56922 (local address 127.23.100.1:38931)
0925 10:40:32.008639 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:32.008646 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:32.008669 (+ 23us) server_negotiation.cc:407] Connection header received
0925 10:40:32.008724 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:32.008729 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:32.008934 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:32.009053 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:32.009896 (+ 843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:40:32.009902 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:40:32.009912 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:40:32.009948 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:40:32.009988 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:40:32.010002 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:40:32.010151 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:32.010254 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 10:40:32.010379 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":262,"threads_started":1}
I20260925 10:40:32.011763 1490 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:40:32.092177 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 1368
I20260925 10:40:32.108654 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 1499
I20260925 10:40:32.112115 1767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:32.109509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55438 (local address 127.23.100.2:43685)
0925 10:40:32.109978 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:32.109991 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:32.110017 (+ 26us) server_negotiation.cc:407] Connection header received
0925 10:40:32.110074 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:32.110080 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:32.110278 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:32.110485 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:32.111425 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:40:32.111431 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:40:32.111441 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:40:32.111457 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:40:32.111493 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:40:32.111506 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:40:32.111646 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:32.111765 (+ 119us) server_negotiation.cc:299] Negotiation successful
0925 10:40:32.111861 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":203,"threads_started":1}
I20260925 10:40:32.113226 1621 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:40:32.192145 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 1499
I20260925 10:40:32.207938 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 1629
I20260925 10:40:32.211862 1769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:32.208867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50818 (local address 127.23.100.3:33941)
0925 10:40:32.209446 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:32.209454 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:32.209483 (+ 29us) server_negotiation.cc:407] Connection header received
0925 10:40:32.209543 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:32.209549 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:32.209750 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:32.209919 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:32.211135 (+ 1216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:40:32.211141 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:40:32.211151 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:40:32.211167 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:40:32.211215 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:40:32.211228 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:40:32.211411 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:32.211483 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 10:40:32.211578 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":280,"threads_started":1}
I20260925 10:40:32.212977 1751 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:40:32.293310 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 1629
I20260925 10:40:32.308933 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 1278
I20260925 10:40:32.310212 1339 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:40:32.410392 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 1278
2026-09-25T10:40:32Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2243 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:40:32 dist-test-slave-dxdd krb5kdc[1774](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:40:32 dist-test-slave-dxdd krb5kdc[1774](info): set up 2 sockets
Sep 25 10:40:32 dist-test-slave-dxdd krb5kdc[1774](info): commencing operation
krb5kdc: starting...
W20260925 10:40:33.491148 23952 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Sep 25 10:40:33 dist-test-slave-dxdd krb5kdc[1774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:40:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:40:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
I20260925 10:40:33.711176 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:39657
--webserver_interface=127.23.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:37423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.62:39657
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:40:34.027638 1787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:40:34.027925 1787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:40:34.028055 1787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:40:34.035025 1787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:40:34.035116 1787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:40:34.035229 1787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:40:34.035326 1787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:40:34.035406 1787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:40:34.044848 1787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:37423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.62:39657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:39657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:40:34.046152 1787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:40:34.047447 1787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:40:34.059106 1793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:40:34.059161 1795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:40:34.059188 1792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:40:34.060050 1787 server_base.cc:1034] running on GCE node
I20260925 10:40:34.060770 1787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:40:34.062064 1787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:40:34.063277 1787 hybrid_clock.cc:648] HybridClock initialized: now 1790332834063218 us; error 46 us; skew 500 ppm
I20260925 10:40:34.063613 1787 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:40:34 dist-test-slave-dxdd krb5kdc[1774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332834, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:40:34.076161 1787 init.cc:377] Logged in from keytab as kudu/127.23.100.62@KRBTEST.COM (short username kudu)
I20260925 10:40:34.077662 1787 webserver.cc:460] Webserver started at http://127.23.100.62:45365/ using document root <none> and password file <none>
I20260925 10:40:34.078423 1787 fs_manager.cc:359] Metadata directory not provided
I20260925 10:40:34.078570 1787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:40:34.078884 1787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:40:34.081779 1787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/master-0/data/instance:
uuid: "749e17b48b13416f85753cb0a1e84d36"
format_stamp: "Formatted at 2026-09-25 10:40:34 on dist-test-slave-dxdd"
server_key: "e2285afa4ff58358caad603406a0eb29"
server_key_iv: "5fe5850939cbc75133f95b6ddd6b70e8"
server_key_version: "encryptionkey@0"
I20260925 10:40:34.082595 1787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/master-0/wal/instance:
uuid: "749e17b48b13416f85753cb0a1e84d36"
format_stamp: "Formatted at 2026-09-25 10:40:34 on dist-test-slave-dxdd"
server_key: "e2285afa4ff58358caad603406a0eb29"
server_key_iv: "5fe5850939cbc75133f95b6ddd6b70e8"
server_key_version: "encryptionkey@0"
I20260925 10:40:34.087530 1787 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 10:40:34.090914 1802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:40:34.092132 1787 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 10:40:34.092314 1787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/master-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/master-0/wal
uuid: "749e17b48b13416f85753cb0a1e84d36"
format_stamp: "Formatted at 2026-09-25 10:40:34 on dist-test-slave-dxdd"
server_key: "e2285afa4ff58358caad603406a0eb29"
server_key_iv: "5fe5850939cbc75133f95b6ddd6b70e8"
server_key_version: "encryptionkey@0"
I20260925 10:40:34.092497 1787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:40:34.123828 1787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:40:34.127601 1787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:40:34.127938 1787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:40:34.145617 1787 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.62:39657
I20260925 10:40:34.145669 1853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.62:39657 every 8 connection(s)
I20260925 10:40:34.146914 1787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
I20260925 10:40:34.150466 1854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:40:34.151798 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 1787
I20260925 10:40:34.152016 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/master-0/wal/instance
I20260925 10:40:34.152840 23952 external_mini_cluster.cc:1428] Setting key c80270d065dfa972e0874a1e2c8ac103
I20260925 10:40:34.164426 1854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 749e17b48b13416f85753cb0a1e84d36: Bootstrap starting.
Sep 25 10:40:34 dist-test-slave-dxdd krb5kdc[1774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:40:34.168109 1854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 749e17b48b13416f85753cb0a1e84d36: Neither blocks nor log segments found. Creating new log.
I20260925 10:40:34.169459 1854 log.cc:824] T 00000000000000000000000000000000 P 749e17b48b13416f85753cb0a1e84d36: Log is configured to *not* fsync() on all Append() calls
I20260925 10:40:34.172631 1854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 749e17b48b13416f85753cb0a1e84d36: No bootstrap required, opened a new log
I20260925 10:40:34.175957 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:34.154798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45958 (local address 127.23.100.62:39657)
0925 10:40:34.155602 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:34.155616 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:34.155657 (+ 41us) server_negotiation.cc:407] Connection header received
0925 10:40:34.157150 (+ 1493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:34.157172 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:34.157527 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:34.157824 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:34.159537 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:34.160758 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:34.162540 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:34.163162 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:34.163288 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:34.167839 (+ 4551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:40:34.167851 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:40:34.167870 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:40:34.167916 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:40:34.171663 (+ 3747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:40:34.172290 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:40:34.172299 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:40:34.172306 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:40:34.172506 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:40:34.172883 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:40:34.172887 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:40:34.172890 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:40:34.173587 (+ 697us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:40:34.173880 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:34.174184 (+ 304us) server_negotiation.cc:299] Negotiation successful
0925 10:40:34.174652 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":390,"threads_started":1}
I20260925 10:40:34.181756 1854 raft_consensus.cc:348] T 00000000000000000000000000000000 P 749e17b48b13416f85753cb0a1e84d36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "749e17b48b13416f85753cb0a1e84d36" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 39657 } }
I20260925 10:40:34.182062 1854 raft_consensus.cc:374] T 00000000000000000000000000000000 P 749e17b48b13416f85753cb0a1e84d36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:40:34.182135 1854 raft_consensus.cc:729] T 00000000000000000000000000000000 P 749e17b48b13416f85753cb0a1e84d36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 749e17b48b13416f85753cb0a1e84d36, State: Initialized, Role: FOLLOWER
I20260925 10:40:34.182616 1854 consensus_queue.cc:260] T 00000000000000000000000000000000 P 749e17b48b13416f85753cb0a1e84d36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "749e17b48b13416f85753cb0a1e84d36" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 39657 } }
I20260925 10:40:34.182802 1854 raft_consensus.cc:388] T 00000000000000000000000000000000 P 749e17b48b13416f85753cb0a1e84d36 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:40:34.182891 1854 raft_consensus.cc:482] T 00000000000000000000000000000000 P 749e17b48b13416f85753cb0a1e84d36 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:40:34.183015 1854 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 749e17b48b13416f85753cb0a1e84d36 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:40:34.184921 1854 raft_consensus.cc:504] T 00000000000000000000000000000000 P 749e17b48b13416f85753cb0a1e84d36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "749e17b48b13416f85753cb0a1e84d36" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 39657 } }
I20260925 10:40:34.185333 1854 leader_election.cc:302] T 00000000000000000000000000000000 P 749e17b48b13416f85753cb0a1e84d36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 749e17b48b13416f85753cb0a1e84d36; no voters:
I20260925 10:40:34.185808 1854 leader_election.cc:288] T 00000000000000000000000000000000 P 749e17b48b13416f85753cb0a1e84d36 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:40:34.185967 1859 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 749e17b48b13416f85753cb0a1e84d36 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:40:34.186653 1859 raft_consensus.cc:686] T 00000000000000000000000000000000 P 749e17b48b13416f85753cb0a1e84d36 [term 1 LEADER]: Becoming Leader. State: Replica: 749e17b48b13416f85753cb0a1e84d36, State: Running, Role: LEADER
I20260925 10:40:34.187204 1859 consensus_queue.cc:237] T 00000000000000000000000000000000 P 749e17b48b13416f85753cb0a1e84d36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "749e17b48b13416f85753cb0a1e84d36" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 39657 } }
I20260925 10:40:34.187924 1854 sys_catalog.cc:564] T 00000000000000000000000000000000 P 749e17b48b13416f85753cb0a1e84d36 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:40:34.190228 1861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 749e17b48b13416f85753cb0a1e84d36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 749e17b48b13416f85753cb0a1e84d36. Latest consensus state: current_term: 1 leader_uuid: "749e17b48b13416f85753cb0a1e84d36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "749e17b48b13416f85753cb0a1e84d36" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 39657 } } }
I20260925 10:40:34.190629 1861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 749e17b48b13416f85753cb0a1e84d36 [sys.catalog]: This master's current role is: LEADER
I20260925 10:40:34.191953 1860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 749e17b48b13416f85753cb0a1e84d36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "749e17b48b13416f85753cb0a1e84d36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "749e17b48b13416f85753cb0a1e84d36" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 39657 } } }
I20260925 10:40:34.192380 1860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 749e17b48b13416f85753cb0a1e84d36 [sys.catalog]: This master's current role is: LEADER
I20260925 10:40:34.192764 1868 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:40:34.198748 1868 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:40:34.208030 1868 catalog_manager.cc:1347] Generated new cluster ID: e8192b3dccd74d288fb66d363e37af25
I20260925 10:40:34.208123 1868 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:40:34.225930 1868 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:40:34.227166 1868 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:40:34.233183 1868 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 749e17b48b13416f85753cb0a1e84d36: Generated new TSK 0
I20260925 10:40:34.233901 1868 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab.
Entry for principal kudu/127.23.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab.
I20260925 10:40:34.292359 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.1:0
--local_ip_for_outbound_sockets=127.23.100.1
--webserver_interface=127.23.100.1
--webserver_port=0
--tserver_master_addrs=127.23.100.62:39657
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab
--principal=kudu/127.23.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:37423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:40:34.610045 1880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:40:34.610358 1880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:40:34.610526 1880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:40:34.617689 1880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:40:34.617833 1880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:40:34.618021 1880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.1
I20260925 10:40:34.627612 1880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:37423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab
--principal=kudu/127.23.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.1
--webserver_port=0
--tserver_master_addrs=127.23.100.62:39657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.1
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:40:34.628895 1880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:40:34.630168 1880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:40:34.643142 1885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:40:34.643186 1886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:40:34.643229 1888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:40:34.644263 1880 server_base.cc:1034] running on GCE node
I20260925 10:40:34.644827 1880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:40:34.645761 1880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:40:34.646977 1880 hybrid_clock.cc:648] HybridClock initialized: now 1790332834646944 us; error 48 us; skew 500 ppm
I20260925 10:40:34.647269 1880 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:40:34 dist-test-slave-dxdd krb5kdc[1774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332834, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:40:34.650168 1880 init.cc:377] Logged in from keytab as kudu/127.23.100.1@KRBTEST.COM (short username kudu)
I20260925 10:40:34.651553 1880 webserver.cc:460] Webserver started at http://127.23.100.1:46217/ using document root <none> and password file <none>
I20260925 10:40:34.652212 1880 fs_manager.cc:359] Metadata directory not provided
I20260925 10:40:34.652345 1880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:40:34.652630 1880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:40:34.655447 1880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/ts-0/data/instance:
uuid: "675bff02c9c9476bacaaa4446d01c5a0"
format_stamp: "Formatted at 2026-09-25 10:40:34 on dist-test-slave-dxdd"
server_key: "9a383b4f2911395d576674d242d8cb55"
server_key_iv: "78855aeb8a9a93adabe731efec545811"
server_key_version: "encryptionkey@0"
I20260925 10:40:34.656162 1880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/ts-0/wal/instance:
uuid: "675bff02c9c9476bacaaa4446d01c5a0"
format_stamp: "Formatted at 2026-09-25 10:40:34 on dist-test-slave-dxdd"
server_key: "9a383b4f2911395d576674d242d8cb55"
server_key_iv: "78855aeb8a9a93adabe731efec545811"
server_key_version: "encryptionkey@0"
I20260925 10:40:34.660995 1880 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 10:40:34.664233 1895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:40:34.665441 1880 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 10:40:34.665596 1880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/ts-0/wal
uuid: "675bff02c9c9476bacaaa4446d01c5a0"
format_stamp: "Formatted at 2026-09-25 10:40:34 on dist-test-slave-dxdd"
server_key: "9a383b4f2911395d576674d242d8cb55"
server_key_iv: "78855aeb8a9a93adabe731efec545811"
server_key_version: "encryptionkey@0"
I20260925 10:40:34.665787 1880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:40:34.688316 1880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:40:34.692255 1880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:40:34.692600 1880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:40:34.693538 1880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:40:34.694869 1880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:40:34.694971 1880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:40:34.695117 1880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:40:34.695186 1880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:40:34.739488 1880 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.1:44967
I20260925 10:40:34.739537 2007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.1:44967 every 8 connection(s)
I20260925 10:40:34.741238 1880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
I20260925 10:40:34.745632 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 1880
I20260925 10:40:34.745887 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/ts-0/wal/instance
I20260925 10:40:34.746757 23952 external_mini_cluster.cc:1428] Setting key b0121165033b13777d4c5ef868f2e17f
Sep 25 10:40:34 dist-test-slave-dxdd krb5kdc[1774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332834, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.1@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:40:34.767787 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:34.743933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:56799 (local address 127.23.100.62:39657)
0925 10:40:34.744401 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:34.744407 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:34.749566 (+ 5159us) server_negotiation.cc:407] Connection header received
0925 10:40:34.749633 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:34.749640 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:34.749723 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:34.749821 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:34.752129 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:34.753410 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:34.755975 (+ 2565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:34.756693 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:34.756837 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:34.761609 (+ 4772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:40:34.761614 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:40:34.761617 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:40:34.761660 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:40:34.764385 (+ 2725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:40:34.765330 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:40:34.765353 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:40:34.765367 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:40:34.765485 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:40:34.766004 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:40:34.766009 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:40:34.766012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:40:34.766367 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:40:34.766508 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:34.767215 (+ 707us) server_negotiation.cc:299] Negotiation successful
0925 10:40:34.767431 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":230,"threads_started":1}
I20260925 10:40:34.768931 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:34.744573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:39657 (local address 127.23.100.1:56799)
0925 10:40:34.746689 (+ 2116us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:34.746741 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:34.749192 (+ 2451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:34.750275 (+ 1083us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:34.750289 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:34.751128 (+ 839us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:34.751944 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:34.751972 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:34.753583 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:34.753591 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:34.755769 (+ 2178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:34.755788 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:34.756937 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:34.756947 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:34.757364 (+ 417us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:34.758208 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:34.758238 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:34.761342 (+ 3104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:40:34.764551 (+ 3209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:40:34.764562 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:40:34.764581 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:40:34.765201 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:40:34.765607 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:40:34.765616 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:40:34.765620 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:40:34.765868 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:40:34.766488 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:40:34.766496 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:40:34.766826 (+ 330us) client_negotiation.cc:769] Sending connection context
0925 10:40:34.767090 (+ 264us) client_negotiation.cc:240] Negotiation successful
0925 10:40:34.767630 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1818,"thread_start_us":336,"threads_started":1}
I20260925 10:40:34.770649 2008 heartbeater.cc:347] Connected to a master server at 127.23.100.62:39657
I20260925 10:40:34.770993 2008 heartbeater.cc:464] Registering TS with master...
I20260925 10:40:34.771878 2008 heartbeater.cc:511] Master 127.23.100.62:39657 requested a full tablet report, sending...
I20260925 10:40:34.774127 1819 ts_manager.cc:194] Registered new tserver with Master: 675bff02c9c9476bacaaa4446d01c5a0 (127.23.100.1:44967)
WARNING: no policy specified for kudu/127.23.100.2@KRBTEST.COM; defaulting to no policy
I20260925 10:40:34.775681 1819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.1@KRBTEST.COM'} at 127.23.100.1:56799
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab.
Entry for principal kudu/127.23.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab.
I20260925 10:40:34.803596 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.2:0
--local_ip_for_outbound_sockets=127.23.100.2
--webserver_interface=127.23.100.2
--webserver_port=0
--tserver_master_addrs=127.23.100.62:39657
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab
--principal=kudu/127.23.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:37423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:40:35.118878 2014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:40:35.119128 2014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:40:35.119314 2014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:40:35.126173 2014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:40:35.126274 2014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:40:35.126502 2014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.2
I20260925 10:40:35.135794 2014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:37423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab
--principal=kudu/127.23.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.2
--webserver_port=0
--tserver_master_addrs=127.23.100.62:39657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.2
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:40:35.137035 2014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:40:35.138299 2014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:40:35.153600 2020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:40:35.153712 2019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:40:35.157644 2022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:40:35.159251 2014 server_base.cc:1034] running on GCE node
I20260925 10:40:35.159937 2014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:40:35.161029 2014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:40:35.162259 2014 hybrid_clock.cc:648] HybridClock initialized: now 1790332835162207 us; error 49 us; skew 500 ppm
I20260925 10:40:35.162627 2014 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:40:35 dist-test-slave-dxdd krb5kdc[1774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332835, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:40:35.165906 2014 init.cc:377] Logged in from keytab as kudu/127.23.100.2@KRBTEST.COM (short username kudu)
I20260925 10:40:35.167335 2014 webserver.cc:460] Webserver started at http://127.23.100.2:39591/ using document root <none> and password file <none>
I20260925 10:40:35.168045 2014 fs_manager.cc:359] Metadata directory not provided
I20260925 10:40:35.168169 2014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:40:35.168499 2014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:40:35.171602 2014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/ts-1/data/instance:
uuid: "98da1065428f4d2785dbd553a4ee7fc7"
format_stamp: "Formatted at 2026-09-25 10:40:35 on dist-test-slave-dxdd"
server_key: "ca9ff92f19dc47b2d716ee7f793ffebb"
server_key_iv: "1f97611270d5a722819a1bb28f83b0ba"
server_key_version: "encryptionkey@0"
I20260925 10:40:35.172324 2014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/ts-1/wal/instance:
uuid: "98da1065428f4d2785dbd553a4ee7fc7"
format_stamp: "Formatted at 2026-09-25 10:40:35 on dist-test-slave-dxdd"
server_key: "ca9ff92f19dc47b2d716ee7f793ffebb"
server_key_iv: "1f97611270d5a722819a1bb28f83b0ba"
server_key_version: "encryptionkey@0"
I20260925 10:40:35.177240 2014 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 10:40:35.180514 2029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:40:35.181749 2014 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 10:40:35.181918 2014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/ts-1/wal
uuid: "98da1065428f4d2785dbd553a4ee7fc7"
format_stamp: "Formatted at 2026-09-25 10:40:35 on dist-test-slave-dxdd"
server_key: "ca9ff92f19dc47b2d716ee7f793ffebb"
server_key_iv: "1f97611270d5a722819a1bb28f83b0ba"
server_key_version: "encryptionkey@0"
I20260925 10:40:35.182195 2014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:40:35.204288 2014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:40:35.208072 2014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:40:35.208351 2014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:40:35.209250 2014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:40:35.210556 2014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:40:35.210628 2014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:40:35.210777 2014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:40:35.210836 2014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:40:35.242951 2014 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.2:37303
I20260925 10:40:35.243014 2141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.2:37303 every 8 connection(s)
I20260925 10:40:35.244243 2014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
I20260925 10:40:35.253439 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 2014
I20260925 10:40:35.253650 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/ts-1/wal/instance
I20260925 10:40:35.254073 23952 external_mini_cluster.cc:1428] Setting key e0b5d30533f66d98fd3cc4555315d491
Sep 25 10:40:35 dist-test-slave-dxdd krb5kdc[1774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332835, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.2@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:40:35.263502 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:35.246748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:46709 (local address 127.23.100.62:39657)
0925 10:40:35.246926 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:35.246930 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:35.248063 (+ 1133us) server_negotiation.cc:407] Connection header received
0925 10:40:35.249922 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:35.249928 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:35.250018 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:35.250123 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:35.251669 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:35.252376 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:35.253949 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:35.254484 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:35.254612 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:35.257898 (+ 3286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:40:35.257904 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:40:35.257909 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:40:35.257948 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:40:35.260522 (+ 2574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:40:35.261208 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:40:35.261215 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:40:35.261218 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:40:35.261350 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:40:35.261776 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:40:35.261782 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:40:35.261785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:40:35.262046 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:40:35.262199 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:35.262933 (+ 734us) server_negotiation.cc:299] Negotiation successful
0925 10:40:35.263150 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 10:40:35.264467 2144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:35.247231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:39657 (local address 127.23.100.2:46709)
0925 10:40:35.247908 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:35.247946 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:35.249658 (+ 1712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:35.250403 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:35.250411 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:35.250959 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:35.251504 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:35.251521 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:35.252524 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:35.252528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:35.253816 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:35.253828 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:35.254697 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:35.254704 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:35.254945 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:35.255630 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:35.255653 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:35.257681 (+ 2028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:40:35.260654 (+ 2973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:40:35.260663 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:40:35.260679 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:40:35.261095 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:40:35.261474 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:40:35.261479 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:40:35.261483 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:40:35.261670 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:40:35.262170 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:40:35.262178 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:40:35.262583 (+ 405us) client_negotiation.cc:769] Sending connection context
0925 10:40:35.262841 (+ 258us) client_negotiation.cc:240] Negotiation successful
0925 10:40:35.263347 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":254,"threads_started":1}
I20260925 10:40:35.265837 2142 heartbeater.cc:347] Connected to a master server at 127.23.100.62:39657
I20260925 10:40:35.266130 2142 heartbeater.cc:464] Registering TS with master...
I20260925 10:40:35.266866 2142 heartbeater.cc:511] Master 127.23.100.62:39657 requested a full tablet report, sending...
I20260925 10:40:35.268343 1819 ts_manager.cc:194] Registered new tserver with Master: 98da1065428f4d2785dbd553a4ee7fc7 (127.23.100.2:37303)
I20260925 10:40:35.269531 1819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.2@KRBTEST.COM'} at 127.23.100.2:46709
WARNING: no policy specified for kudu/127.23.100.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab.
Entry for principal kudu/127.23.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab.
I20260925 10:40:35.303038 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.3:0
--local_ip_for_outbound_sockets=127.23.100.3
--webserver_interface=127.23.100.3
--webserver_port=0
--tserver_master_addrs=127.23.100.62:39657
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab
--principal=kudu/127.23.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:37423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:40:35.617465 2147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:40:35.617715 2147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:40:35.617802 2147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:40:35.624678 2147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:40:35.624773 2147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:40:35.624891 2147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.3
I20260925 10:40:35.634533 2147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:37423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab
--principal=kudu/127.23.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.3
--webserver_port=0
--tserver_master_addrs=127.23.100.62:39657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.3
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:40:35.635670 2147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:40:35.636890 2147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:40:35.649449 2152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:40:35.649458 2153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:40:35.649451 2155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:40:35.650211 2147 server_base.cc:1034] running on GCE node
I20260925 10:40:35.650765 2147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:40:35.651582 2147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:40:35.652791 2147 hybrid_clock.cc:648] HybridClock initialized: now 1790332835652751 us; error 33 us; skew 500 ppm
I20260925 10:40:35.653074 2147 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:40:35 dist-test-slave-dxdd krb5kdc[1774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332835, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:40:35.656097 2147 init.cc:377] Logged in from keytab as kudu/127.23.100.3@KRBTEST.COM (short username kudu)
I20260925 10:40:35.657389 2147 webserver.cc:460] Webserver started at http://127.23.100.3:44437/ using document root <none> and password file <none>
I20260925 10:40:35.657979 2147 fs_manager.cc:359] Metadata directory not provided
I20260925 10:40:35.658067 2147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:40:35.658288 2147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:40:35.660981 2147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/ts-2/data/instance:
uuid: "642c22b4cc6342838873833f4d92747b"
format_stamp: "Formatted at 2026-09-25 10:40:35 on dist-test-slave-dxdd"
server_key: "3c10da31b9bddacd3c3f8b6b1da45e91"
server_key_iv: "5092236d208fff67731a910dc4893863"
server_key_version: "encryptionkey@0"
I20260925 10:40:35.661625 2147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/ts-2/wal/instance:
uuid: "642c22b4cc6342838873833f4d92747b"
format_stamp: "Formatted at 2026-09-25 10:40:35 on dist-test-slave-dxdd"
server_key: "3c10da31b9bddacd3c3f8b6b1da45e91"
server_key_iv: "5092236d208fff67731a910dc4893863"
server_key_version: "encryptionkey@0"
I20260925 10:40:35.666714 2147 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 10:40:35.669906 2162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:40:35.671192 2147 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:40:35.671357 2147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/ts-2/wal
uuid: "642c22b4cc6342838873833f4d92747b"
format_stamp: "Formatted at 2026-09-25 10:40:35 on dist-test-slave-dxdd"
server_key: "3c10da31b9bddacd3c3f8b6b1da45e91"
server_key_iv: "5092236d208fff67731a910dc4893863"
server_key_version: "encryptionkey@0"
I20260925 10:40:35.671548 2147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:40:35.681957 2147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:40:35.686028 2147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:40:35.686349 2147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:40:35.687245 2147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:40:35.688530 2147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:40:35.688616 2147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:40:35.688736 2147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:40:35.688804 2147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:40:35.719660 2147 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.3:36099
I20260925 10:40:35.719691 2274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.3:36099 every 8 connection(s)
I20260925 10:40:35.720935 2147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
I20260925 10:40:35.723310 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 2147
I20260925 10:40:35.723528 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/ts-2/wal/instance
I20260925 10:40:35.723992 23952 external_mini_cluster.cc:1428] Setting key 163af01b9397f0e71615a141378e74bb
Sep 25 10:40:35 dist-test-slave-dxdd krb5kdc[1774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332835, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.3@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:40:35.739214 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:35.723366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:46465 (local address 127.23.100.62:39657)
0925 10:40:35.723558 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:35.723564 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:35.724718 (+ 1154us) server_negotiation.cc:407] Connection header received
0925 10:40:35.726531 (+ 1813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:35.726537 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:35.726654 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:35.726774 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:35.728115 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:35.728823 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:35.730177 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:35.730629 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:35.730719 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:35.733632 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:40:35.733637 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:40:35.733640 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:40:35.733671 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:40:35.736184 (+ 2513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:40:35.736750 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:40:35.736754 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:40:35.736756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:40:35.736886 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:40:35.737254 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:40:35.737259 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:40:35.737262 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:40:35.737605 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:40:35.737762 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:35.738567 (+ 805us) server_negotiation.cc:299] Negotiation successful
0925 10:40:35.738791 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 10:40:35.740156 2277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:35.723823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:39657 (local address 127.23.100.3:46465)
0925 10:40:35.724571 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:35.724612 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:35.726245 (+ 1633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:35.726991 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:35.726999 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:35.727466 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:40:35.727962 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:35.727978 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:35.728951 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:35.728954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:35.730064 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:35.730078 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:35.730789 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:35.730794 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:35.730972 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:35.731534 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:40:35.731553 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:40:35.733469 (+ 1916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:40:35.736283 (+ 2814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:40:35.736288 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:40:35.736298 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:40:35.736661 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:40:35.736972 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:40:35.736976 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:40:35.736978 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:40:35.737127 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:40:35.737745 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:40:35.737752 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:40:35.738115 (+ 363us) client_negotiation.cc:769] Sending connection context
0925 10:40:35.738462 (+ 347us) client_negotiation.cc:240] Negotiation successful
0925 10:40:35.739075 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":323,"threads_started":1}
I20260925 10:40:35.741442 2275 heartbeater.cc:347] Connected to a master server at 127.23.100.62:39657
I20260925 10:40:35.741767 2275 heartbeater.cc:464] Registering TS with master...
I20260925 10:40:35.742475 2275 heartbeater.cc:511] Master 127.23.100.62:39657 requested a full tablet report, sending...
I20260925 10:40:35.743736 1819 ts_manager.cc:194] Registered new tserver with Master: 642c22b4cc6342838873833f4d92747b (127.23.100.3:36099)
I20260925 10:40:35.744442 1819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.3@KRBTEST.COM'} at 127.23.100.3:46465
I20260925 10:40:35.751075 23952 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:40:35.766233 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:35.755140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45964 (local address 127.23.100.62:39657)
0925 10:40:35.755346 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:35.755354 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:35.755683 (+ 329us) server_negotiation.cc:407] Connection header received
0925 10:40:35.755812 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:35.755815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:35.755870 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:35.755950 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:35.757144 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:35.757806 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:35.758985 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:35.759367 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:35.759456 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:35.761684 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:40:35.761689 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:40:35.761692 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:40:35.761719 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:40:35.764019 (+ 2300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:40:35.764560 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:40:35.764563 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:40:35.764565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:40:35.764677 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:40:35.764986 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:40:35.764989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:40:35.764991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:40:35.765262 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:40:35.765433 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:35.765776 (+ 343us) server_negotiation.cc:299] Negotiation successful
0925 10:40:35.765913 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 10:40:35.769877 1819 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45964:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:40:35.776513 1819 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:40:35.778765 2008 heartbeater.cc:503] Master 127.23.100.62:39657 was elected leader, sending a full tablet report...
I20260925 10:40:35.799270 2290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:35.788822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41458 (local address 127.23.100.1:44967)
0925 10:40:35.790644 (+ 1822us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:35.790655 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:35.791540 (+ 885us) server_negotiation.cc:407] Connection header received
0925 10:40:35.791706 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:35.791713 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:35.791965 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:35.792157 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:40:35.793182 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:35.794774 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:35.797305 (+ 2531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:35.798197 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:35.798352 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:35.798579 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:35.798804 (+ 225us) server_negotiation.cc:299] Negotiation successful
0925 10:40:35.798950 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1597,"thread_start_us":1405,"threads_started":1}
I20260925 10:40:35.799345 2287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:35.788098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:44967 (local address 127.0.0.1:41458)
0925 10:40:35.791439 (+ 3341us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:35.791452 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:35.791575 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:35.792395 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:35.792402 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:35.792430 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:40:35.792925 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:35.792940 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:35.794908 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:35.794912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:35.797166 (+ 2254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:35.797176 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:35.798395 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:35.798402 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:35.798577 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:35.798636 (+ 59us) client_negotiation.cc:769] Sending connection context
0925 10:40:35.798757 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 10:40:35.798940 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3208,"mutex_wait_us":29,"thread_start_us":1277,"threads_started":1}
I20260925 10:40:35.799942 2288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:35.788026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37738 (local address 127.23.100.2:37303)
0925 10:40:35.789174 (+ 1148us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:35.789186 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:35.790005 (+ 819us) server_negotiation.cc:407] Connection header received
0925 10:40:35.791119 (+ 1114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:35.791130 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:35.791398 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:35.791601 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:40:35.792821 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:35.794341 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:35.797004 (+ 2663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:35.798459 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:35.798581 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:35.798760 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:35.799434 (+ 674us) server_negotiation.cc:299] Negotiation successful
0925 10:40:35.799587 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":939,"thread_start_us":761,"threads_started":1}
I20260925 10:40:35.800896 2289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:35.787816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.2:37303 (local address 127.0.0.1:37738)
0925 10:40:35.789826 (+ 2010us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:35.789884 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:35.790835 (+ 951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:35.791899 (+ 1064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:35.791906 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:35.791954 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:40:35.792594 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:35.792613 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:35.794484 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:35.794489 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:35.796812 (+ 2323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:35.796829 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:35.798896 (+ 2067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:35.798903 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:35.799079 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:35.799111 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 10:40:35.799199 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 10:40:35.799322 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1709,"mutex_wait_us":48,"thread_start_us":3354,"threads_started":1}
I20260925 10:40:35.801373 2291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:35.789743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.3:36099 (local address 127.0.0.1:60492)
0925 10:40:35.792909 (+ 3166us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:35.792925 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:35.793040 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:35.793747 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:35.793752 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:35.793777 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:40:35.794130 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:35.794140 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:35.796027 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:35.796032 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:35.798241 (+ 2209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:35.798256 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:35.800408 (+ 2152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:35.800414 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:35.800604 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:35.800637 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 10:40:35.800715 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 10:40:35.800848 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3022,"mutex_wait_us":3,"thread_start_us":2831,"threads_started":1}
I20260925 10:40:35.803138 2292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:35.790441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60492 (local address 127.23.100.3:36099)
0925 10:40:35.792168 (+ 1727us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:35.792176 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:35.793146 (+ 970us) server_negotiation.cc:407] Connection header received
0925 10:40:35.793311 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:35.793321 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:35.793546 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:35.793743 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:40:35.794364 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:35.795880 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:35.798491 (+ 2611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:35.800207 (+ 1716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:35.800348 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:35.802605 (+ 2257us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:35.802678 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 10:40:35.802817 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1567,"thread_start_us":1415,"threads_started":1}
I20260925 10:40:35.803094 1943 tablet_service.cc:1467] Processing CreateTablet for tablet 1b068e782424424287d236c2a539568b (DEFAULT_TABLE table=test-table [id=384e0ea1d4c34fddba45a9bb6b2b1c43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:40:35.803913 2077 tablet_service.cc:1467] Processing CreateTablet for tablet 1b068e782424424287d236c2a539568b (DEFAULT_TABLE table=test-table [id=384e0ea1d4c34fddba45a9bb6b2b1c43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:40:35.804322 1943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b068e782424424287d236c2a539568b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:40:35.804885 2077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b068e782424424287d236c2a539568b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:40:35.813374 2210 tablet_service.cc:1467] Processing CreateTablet for tablet 1b068e782424424287d236c2a539568b (DEFAULT_TABLE table=test-table [id=384e0ea1d4c34fddba45a9bb6b2b1c43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:40:35.814680 2210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b068e782424424287d236c2a539568b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:40:35.814954 2293 tablet_bootstrap.cc:492] T 1b068e782424424287d236c2a539568b P 675bff02c9c9476bacaaa4446d01c5a0: Bootstrap starting.
I20260925 10:40:35.816702 2294 tablet_bootstrap.cc:492] T 1b068e782424424287d236c2a539568b P 98da1065428f4d2785dbd553a4ee7fc7: Bootstrap starting.
I20260925 10:40:35.819173 2293 tablet_bootstrap.cc:654] T 1b068e782424424287d236c2a539568b P 675bff02c9c9476bacaaa4446d01c5a0: Neither blocks nor log segments found. Creating new log.
I20260925 10:40:35.819993 2294 tablet_bootstrap.cc:654] T 1b068e782424424287d236c2a539568b P 98da1065428f4d2785dbd553a4ee7fc7: Neither blocks nor log segments found. Creating new log.
I20260925 10:40:35.820465 2293 log.cc:824] T 1b068e782424424287d236c2a539568b P 675bff02c9c9476bacaaa4446d01c5a0: Log is configured to *not* fsync() on all Append() calls
I20260925 10:40:35.821183 2294 log.cc:824] T 1b068e782424424287d236c2a539568b P 98da1065428f4d2785dbd553a4ee7fc7: Log is configured to *not* fsync() on all Append() calls
I20260925 10:40:35.823426 2293 tablet_bootstrap.cc:492] T 1b068e782424424287d236c2a539568b P 675bff02c9c9476bacaaa4446d01c5a0: No bootstrap required, opened a new log
I20260925 10:40:35.823685 2293 ts_tablet_manager.cc:1397] T 1b068e782424424287d236c2a539568b P 675bff02c9c9476bacaaa4446d01c5a0: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260925 10:40:35.824600 2294 tablet_bootstrap.cc:492] T 1b068e782424424287d236c2a539568b P 98da1065428f4d2785dbd553a4ee7fc7: No bootstrap required, opened a new log
I20260925 10:40:35.824841 2294 ts_tablet_manager.cc:1397] T 1b068e782424424287d236c2a539568b P 98da1065428f4d2785dbd553a4ee7fc7: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260925 10:40:35.826112 2297 tablet_bootstrap.cc:492] T 1b068e782424424287d236c2a539568b P 642c22b4cc6342838873833f4d92747b: Bootstrap starting.
I20260925 10:40:35.828992 2297 tablet_bootstrap.cc:654] T 1b068e782424424287d236c2a539568b P 642c22b4cc6342838873833f4d92747b: Neither blocks nor log segments found. Creating new log.
I20260925 10:40:35.829851 2297 log.cc:824] T 1b068e782424424287d236c2a539568b P 642c22b4cc6342838873833f4d92747b: Log is configured to *not* fsync() on all Append() calls
I20260925 10:40:35.832023 2297 tablet_bootstrap.cc:492] T 1b068e782424424287d236c2a539568b P 642c22b4cc6342838873833f4d92747b: No bootstrap required, opened a new log
I20260925 10:40:35.832271 2297 ts_tablet_manager.cc:1397] T 1b068e782424424287d236c2a539568b P 642c22b4cc6342838873833f4d92747b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 10:40:35.834837 2294 raft_consensus.cc:348] T 1b068e782424424287d236c2a539568b P 98da1065428f4d2785dbd553a4ee7fc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98da1065428f4d2785dbd553a4ee7fc7" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 37303 } } peers { permanent_uuid: "675bff02c9c9476bacaaa4446d01c5a0" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 44967 } } peers { permanent_uuid: "642c22b4cc6342838873833f4d92747b" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 36099 } }
I20260925 10:40:35.834839 2293 raft_consensus.cc:348] T 1b068e782424424287d236c2a539568b P 675bff02c9c9476bacaaa4446d01c5a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98da1065428f4d2785dbd553a4ee7fc7" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 37303 } } peers { permanent_uuid: "675bff02c9c9476bacaaa4446d01c5a0" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 44967 } } peers { permanent_uuid: "642c22b4cc6342838873833f4d92747b" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 36099 } }
I20260925 10:40:35.835256 2294 raft_consensus.cc:374] T 1b068e782424424287d236c2a539568b P 98da1065428f4d2785dbd553a4ee7fc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:40:35.835256 2293 raft_consensus.cc:374] T 1b068e782424424287d236c2a539568b P 675bff02c9c9476bacaaa4446d01c5a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:40:35.835354 2294 raft_consensus.cc:729] T 1b068e782424424287d236c2a539568b P 98da1065428f4d2785dbd553a4ee7fc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98da1065428f4d2785dbd553a4ee7fc7, State: Initialized, Role: FOLLOWER
I20260925 10:40:35.835366 2293 raft_consensus.cc:729] T 1b068e782424424287d236c2a539568b P 675bff02c9c9476bacaaa4446d01c5a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 675bff02c9c9476bacaaa4446d01c5a0, State: Initialized, Role: FOLLOWER
I20260925 10:40:35.835955 2293 consensus_queue.cc:260] T 1b068e782424424287d236c2a539568b P 675bff02c9c9476bacaaa4446d01c5a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98da1065428f4d2785dbd553a4ee7fc7" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 37303 } } peers { permanent_uuid: "675bff02c9c9476bacaaa4446d01c5a0" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 44967 } } peers { permanent_uuid: "642c22b4cc6342838873833f4d92747b" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 36099 } }
I20260925 10:40:35.835968 2294 consensus_queue.cc:260] T 1b068e782424424287d236c2a539568b P 98da1065428f4d2785dbd553a4ee7fc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98da1065428f4d2785dbd553a4ee7fc7" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 37303 } } peers { permanent_uuid: "675bff02c9c9476bacaaa4446d01c5a0" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 44967 } } peers { permanent_uuid: "642c22b4cc6342838873833f4d92747b" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 36099 } }
I20260925 10:40:35.837867 2294 ts_tablet_manager.cc:1428] T 1b068e782424424287d236c2a539568b P 98da1065428f4d2785dbd553a4ee7fc7: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260925 10:40:35.839280 2293 ts_tablet_manager.cc:1428] T 1b068e782424424287d236c2a539568b P 675bff02c9c9476bacaaa4446d01c5a0: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260925 10:40:35.839679 2142 heartbeater.cc:503] Master 127.23.100.62:39657 was elected leader, sending a full tablet report...
I20260925 10:40:35.840224 2297 raft_consensus.cc:348] T 1b068e782424424287d236c2a539568b P 642c22b4cc6342838873833f4d92747b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98da1065428f4d2785dbd553a4ee7fc7" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 37303 } } peers { permanent_uuid: "675bff02c9c9476bacaaa4446d01c5a0" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 44967 } } peers { permanent_uuid: "642c22b4cc6342838873833f4d92747b" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 36099 } }
I20260925 10:40:35.840644 2297 raft_consensus.cc:374] T 1b068e782424424287d236c2a539568b P 642c22b4cc6342838873833f4d92747b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:40:35.840736 2297 raft_consensus.cc:729] T 1b068e782424424287d236c2a539568b P 642c22b4cc6342838873833f4d92747b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 642c22b4cc6342838873833f4d92747b, State: Initialized, Role: FOLLOWER
I20260925 10:40:35.841403 2297 consensus_queue.cc:260] T 1b068e782424424287d236c2a539568b P 642c22b4cc6342838873833f4d92747b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98da1065428f4d2785dbd553a4ee7fc7" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 37303 } } peers { permanent_uuid: "675bff02c9c9476bacaaa4446d01c5a0" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 44967 } } peers { permanent_uuid: "642c22b4cc6342838873833f4d92747b" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 36099 } }
I20260925 10:40:35.842564 2275 heartbeater.cc:503] Master 127.23.100.62:39657 was elected leader, sending a full tablet report...
I20260925 10:40:35.843201 2297 ts_tablet_manager.cc:1428] T 1b068e782424424287d236c2a539568b P 642c22b4cc6342838873833f4d92747b: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
W20260925 10:40:35.972658 2276 tablet.cc:2369] T 1b068e782424424287d236c2a539568b P 642c22b4cc6342838873833f4d92747b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:40:35.993841 2009 tablet.cc:2369] T 1b068e782424424287d236c2a539568b P 675bff02c9c9476bacaaa4446d01c5a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:40:35.996107 2143 tablet.cc:2369] T 1b068e782424424287d236c2a539568b P 98da1065428f4d2785dbd553a4ee7fc7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:40:36.041118 2301 raft_consensus.cc:482] T 1b068e782424424287d236c2a539568b P 642c22b4cc6342838873833f4d92747b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:40:36.041390 2301 raft_consensus.cc:504] T 1b068e782424424287d236c2a539568b P 642c22b4cc6342838873833f4d92747b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98da1065428f4d2785dbd553a4ee7fc7" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 37303 } } peers { permanent_uuid: "675bff02c9c9476bacaaa4446d01c5a0" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 44967 } } peers { permanent_uuid: "642c22b4cc6342838873833f4d92747b" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 36099 } }
I20260925 10:40:36.043184 2301 leader_election.cc:288] T 1b068e782424424287d236c2a539568b P 642c22b4cc6342838873833f4d92747b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98da1065428f4d2785dbd553a4ee7fc7 (127.23.100.2:37303), 675bff02c9c9476bacaaa4446d01c5a0 (127.23.100.1:44967)
I20260925 10:40:36.048983 2277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:36.043460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:44967 (local address 127.23.100.3:40577)
0925 10:40:36.043706 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:36.043745 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:36.043885 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:36.044509 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:36.044514 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:36.044540 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:40:36.044715 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:36.044723 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:36.045831 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:36.045838 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:36.047185 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:36.047196 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:36.048254 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:36.048260 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:36.048443 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:36.048466 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 10:40:36.048567 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 10:40:36.048707 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260925 10:40:36.048974 2290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:36.043920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:40577 (local address 127.23.100.1:44967)
0925 10:40:36.044175 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:36.044181 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:36.044200 (+ 19us) server_negotiation.cc:407] Connection header received
0925 10:40:36.044278 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:36.044283 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:36.044361 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:36.044490 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:40:36.044851 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:36.045690 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:36.047336 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:36.048108 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:36.048279 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:36.048353 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:36.048592 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 10:40:36.048727 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 10:40:36.049449 2288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:36.043591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:55251 (local address 127.23.100.2:37303)
0925 10:40:36.043803 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:36.043809 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:36.044579 (+ 770us) server_negotiation.cc:407] Connection header received
0925 10:40:36.044813 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:36.044817 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:36.044878 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:36.044968 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:40:36.045499 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:36.046185 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:36.047810 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:36.048578 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:36.048753 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:36.048819 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:36.049074 (+ 255us) server_negotiation.cc:299] Negotiation successful
0925 10:40:36.049193 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 10:40:36.049542 2302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:36.043574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.2:37303 (local address 127.23.100.3:55251)
0925 10:40:36.044485 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0925 10:40:36.044498 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:40:36.044678 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:40:36.044982 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:40:36.044986 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:40:36.045009 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:40:36.045367 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:36.045375 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:36.046347 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:36.046353 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:36.047667 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:40:36.047678 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:40:36.048775 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:40:36.048781 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:40:36.048933 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:36.048953 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 10:40:36.049050 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 10:40:36.049164 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":818,"spinlock_wait_cycles":17280,"thread_start_us":558,"threads_started":1}
I20260925 10:40:36.050231 1963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b068e782424424287d236c2a539568b" candidate_uuid: "642c22b4cc6342838873833f4d92747b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "675bff02c9c9476bacaaa4446d01c5a0" is_pre_election: true
I20260925 10:40:36.050472 2097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b068e782424424287d236c2a539568b" candidate_uuid: "642c22b4cc6342838873833f4d92747b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98da1065428f4d2785dbd553a4ee7fc7" is_pre_election: true
I20260925 10:40:36.050853 1963 raft_consensus.cc:2454] T 1b068e782424424287d236c2a539568b P 675bff02c9c9476bacaaa4446d01c5a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 642c22b4cc6342838873833f4d92747b in term 0.
I20260925 10:40:36.050981 2097 raft_consensus.cc:2454] T 1b068e782424424287d236c2a539568b P 98da1065428f4d2785dbd553a4ee7fc7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 642c22b4cc6342838873833f4d92747b in term 0.
I20260925 10:40:36.051499 2163 leader_election.cc:302] T 1b068e782424424287d236c2a539568b P 642c22b4cc6342838873833f4d92747b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 642c22b4cc6342838873833f4d92747b, 675bff02c9c9476bacaaa4446d01c5a0; no voters:
I20260925 10:40:36.051867 2301 raft_consensus.cc:2781] T 1b068e782424424287d236c2a539568b P 642c22b4cc6342838873833f4d92747b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:40:36.052016 2301 raft_consensus.cc:482] T 1b068e782424424287d236c2a539568b P 642c22b4cc6342838873833f4d92747b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:40:36.052125 2301 raft_consensus.cc:3037] T 1b068e782424424287d236c2a539568b P 642c22b4cc6342838873833f4d92747b [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:40:36.054363 2301 raft_consensus.cc:504] T 1b068e782424424287d236c2a539568b P 642c22b4cc6342838873833f4d92747b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98da1065428f4d2785dbd553a4ee7fc7" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 37303 } } peers { permanent_uuid: "675bff02c9c9476bacaaa4446d01c5a0" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 44967 } } peers { permanent_uuid: "642c22b4cc6342838873833f4d92747b" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 36099 } }
I20260925 10:40:36.055013 2301 leader_election.cc:288] T 1b068e782424424287d236c2a539568b P 642c22b4cc6342838873833f4d92747b [CANDIDATE]: Term 1 election: Requested vote from peers 98da1065428f4d2785dbd553a4ee7fc7 (127.23.100.2:37303), 675bff02c9c9476bacaaa4446d01c5a0 (127.23.100.1:44967)
I20260925 10:40:36.055405 2097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b068e782424424287d236c2a539568b" candidate_uuid: "642c22b4cc6342838873833f4d92747b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98da1065428f4d2785dbd553a4ee7fc7"
I20260925 10:40:36.055456 1963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b068e782424424287d236c2a539568b" candidate_uuid: "642c22b4cc6342838873833f4d92747b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "675bff02c9c9476bacaaa4446d01c5a0"
I20260925 10:40:36.055635 2097 raft_consensus.cc:3037] T 1b068e782424424287d236c2a539568b P 98da1065428f4d2785dbd553a4ee7fc7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:40:36.055655 1963 raft_consensus.cc:3037] T 1b068e782424424287d236c2a539568b P 675bff02c9c9476bacaaa4446d01c5a0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:40:36.058854 1963 raft_consensus.cc:2454] T 1b068e782424424287d236c2a539568b P 675bff02c9c9476bacaaa4446d01c5a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 642c22b4cc6342838873833f4d92747b in term 1.
I20260925 10:40:36.058854 2097 raft_consensus.cc:2454] T 1b068e782424424287d236c2a539568b P 98da1065428f4d2785dbd553a4ee7fc7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 642c22b4cc6342838873833f4d92747b in term 1.
I20260925 10:40:36.059263 2165 leader_election.cc:302] T 1b068e782424424287d236c2a539568b P 642c22b4cc6342838873833f4d92747b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 642c22b4cc6342838873833f4d92747b, 98da1065428f4d2785dbd553a4ee7fc7; no voters:
I20260925 10:40:36.059612 2301 raft_consensus.cc:2781] T 1b068e782424424287d236c2a539568b P 642c22b4cc6342838873833f4d92747b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:40:36.060055 2301 raft_consensus.cc:686] T 1b068e782424424287d236c2a539568b P 642c22b4cc6342838873833f4d92747b [term 1 LEADER]: Becoming Leader. State: Replica: 642c22b4cc6342838873833f4d92747b, State: Running, Role: LEADER
I20260925 10:40:36.060508 2301 consensus_queue.cc:237] T 1b068e782424424287d236c2a539568b P 642c22b4cc6342838873833f4d92747b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98da1065428f4d2785dbd553a4ee7fc7" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 37303 } } peers { permanent_uuid: "675bff02c9c9476bacaaa4446d01c5a0" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 44967 } } peers { permanent_uuid: "642c22b4cc6342838873833f4d92747b" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 36099 } }
I20260925 10:40:36.065259 1819 catalog_manager.cc:5495] T 1b068e782424424287d236c2a539568b P 642c22b4cc6342838873833f4d92747b reported cstate change: term changed from 0 to 1, leader changed from <none> to 642c22b4cc6342838873833f4d92747b (127.23.100.3). New cstate: current_term: 1 leader_uuid: "642c22b4cc6342838873833f4d92747b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98da1065428f4d2785dbd553a4ee7fc7" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 37303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "675bff02c9c9476bacaaa4446d01c5a0" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 44967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "642c22b4cc6342838873833f4d92747b" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 36099 } health_report { overall_health: HEALTHY } } }
I20260925 10:40:36.090742 2292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:36.085723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60496 (local address 127.23.100.3:36099)
0925 10:40:36.085965 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:36.085972 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:36.086002 (+ 30us) server_negotiation.cc:407] Connection header received
0925 10:40:36.086204 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:36.086208 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:36.086286 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:36.086469 (+ 183us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:40:36.086832 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:36.087519 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:36.088799 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:36.089162 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:36.089250 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:36.089681 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:40:36.090156 (+ 475us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:40:36.090264 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:36.090403 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 10:40:36.090465 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260925 10:40:36.098752 2097 raft_consensus.cc:1261] T 1b068e782424424287d236c2a539568b P 98da1065428f4d2785dbd553a4ee7fc7 [term 1 FOLLOWER]: Refusing update from remote peer 642c22b4cc6342838873833f4d92747b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:40:36.098755 1963 raft_consensus.cc:1261] T 1b068e782424424287d236c2a539568b P 675bff02c9c9476bacaaa4446d01c5a0 [term 1 FOLLOWER]: Refusing update from remote peer 642c22b4cc6342838873833f4d92747b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:40:36.099798 2303 consensus_queue.cc:1035] T 1b068e782424424287d236c2a539568b P 642c22b4cc6342838873833f4d92747b [LEADER]: Connected to new peer: Peer: permanent_uuid: "675bff02c9c9476bacaaa4446d01c5a0" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 44967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:40:36.100142 2301 consensus_queue.cc:1035] T 1b068e782424424287d236c2a539568b P 642c22b4cc6342838873833f4d92747b [LEADER]: Connected to new peer: Peer: permanent_uuid: "98da1065428f4d2785dbd553a4ee7fc7" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 37303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:40:36.122670 1818 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:40:36.122961 1818 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:40:36.126421 1818 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 749e17b48b13416f85753cb0a1e84d36: Sending DeleteTablet for 3 replicas of tablet 1b068e782424424287d236c2a539568b
I20260925 10:40:36.127578 1943 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b068e782424424287d236c2a539568b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:40:36 UTC) from {username='kudu', principal='kudu/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:41458
I20260925 10:40:36.127561 2077 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b068e782424424287d236c2a539568b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:40:36 UTC) from {username='kudu', principal='kudu/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:37738
I20260925 10:40:36.128118 2210 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b068e782424424287d236c2a539568b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:40:36 UTC) from {username='kudu', principal='kudu/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:60492
I20260925 10:40:36.129731 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 1880
I20260925 10:40:36.165267 2318 tablet_replica.cc:330] T 1b068e782424424287d236c2a539568b P 642c22b4cc6342838873833f4d92747b: stopping tablet replica
I20260925 10:40:36.165652 2316 tablet_replica.cc:330] T 1b068e782424424287d236c2a539568b P 675bff02c9c9476bacaaa4446d01c5a0: stopping tablet replica
I20260925 10:40:36.166759 2318 raft_consensus.cc:2229] T 1b068e782424424287d236c2a539568b P 642c22b4cc6342838873833f4d92747b [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:40:36.166322 2316 raft_consensus.cc:2229] T 1b068e782424424287d236c2a539568b P 675bff02c9c9476bacaaa4446d01c5a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:40:36.170199 2316 raft_consensus.cc:2258] T 1b068e782424424287d236c2a539568b P 675bff02c9c9476bacaaa4446d01c5a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:40:36.171006 2318 raft_consensus.cc:2258] T 1b068e782424424287d236c2a539568b P 642c22b4cc6342838873833f4d92747b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:40:36.173584 2317 tablet_replica.cc:330] T 1b068e782424424287d236c2a539568b P 98da1065428f4d2785dbd553a4ee7fc7: stopping tablet replica
I20260925 10:40:36.174082 2317 raft_consensus.cc:2229] T 1b068e782424424287d236c2a539568b P 98da1065428f4d2785dbd553a4ee7fc7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:40:36.174633 2317 raft_consensus.cc:2258] T 1b068e782424424287d236c2a539568b P 98da1065428f4d2785dbd553a4ee7fc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:40:36.175209 2318 ts_tablet_manager.cc:1905] T 1b068e782424424287d236c2a539568b P 642c22b4cc6342838873833f4d92747b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:40:36.176438 2316 ts_tablet_manager.cc:1905] T 1b068e782424424287d236c2a539568b P 675bff02c9c9476bacaaa4446d01c5a0: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 25 10:40:36 dist-test-slave-dxdd krb5kdc[1774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.1@KRBTEST.COM
I20260925 10:40:36.178049 2317 ts_tablet_manager.cc:1905] T 1b068e782424424287d236c2a539568b P 98da1065428f4d2785dbd553a4ee7fc7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:40:36.183220 2318 ts_tablet_manager.cc:1918] T 1b068e782424424287d236c2a539568b P 642c22b4cc6342838873833f4d92747b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:40:36.183389 2318 log.cc:1196] T 1b068e782424424287d236c2a539568b P 642c22b4cc6342838873833f4d92747b: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/ts-2/wal/wals/1b068e782424424287d236c2a539568b
I20260925 10:40:36.183905 2318 ts_tablet_manager.cc:1939] T 1b068e782424424287d236c2a539568b P 642c22b4cc6342838873833f4d92747b: Deleting consensus metadata
I20260925 10:40:36.185081 2316 ts_tablet_manager.cc:1918] T 1b068e782424424287d236c2a539568b P 675bff02c9c9476bacaaa4446d01c5a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:40:36.185247 2316 log.cc:1196] T 1b068e782424424287d236c2a539568b P 675bff02c9c9476bacaaa4446d01c5a0: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/ts-0/wal/wals/1b068e782424424287d236c2a539568b
I20260925 10:40:36.185758 1806 catalog_manager.cc:4841] TS 642c22b4cc6342838873833f4d92747b (127.23.100.3:36099): tablet 1b068e782424424287d236c2a539568b (table test-table [id=384e0ea1d4c34fddba45a9bb6b2b1c43]) successfully deleted
I20260925 10:40:36.186779 2290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:36.131312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41464 (local address 127.23.100.1:44967)
0925 10:40:36.131556 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:36.131565 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:36.155997 (+ 24432us) server_negotiation.cc:407] Connection header received
0925 10:40:36.156576 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:36.156586 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:36.156696 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:36.156787 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:36.159062 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:36.160656 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:36.167267 (+ 6611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:36.168196 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:36.168323 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:36.177978 (+ 9655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:40:36.177991 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:40:36.178008 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:40:36.178052 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:40:36.182142 (+ 4090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:40:36.183034 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:40:36.183043 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:40:36.183047 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:40:36.183221 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:40:36.183847 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:40:36.183853 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:40:36.183856 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:40:36.184386 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:40:36.184697 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:36.185419 (+ 722us) server_negotiation.cc:299] Negotiation successful
0925 10:40:36.185660 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 10:40:36.186954 2316 ts_tablet_manager.cc:1939] T 1b068e782424424287d236c2a539568b P 675bff02c9c9476bacaaa4446d01c5a0: Deleting consensus metadata
I20260925 10:40:36.187402 2003 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:40:36.193614 2317 ts_tablet_manager.cc:1918] T 1b068e782424424287d236c2a539568b P 98da1065428f4d2785dbd553a4ee7fc7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:40:36.193774 2317 log.cc:1196] T 1b068e782424424287d236c2a539568b P 98da1065428f4d2785dbd553a4ee7fc7: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790332740062128-23952-0/minicluster-data/ts-1/wal/wals/1b068e782424424287d236c2a539568b
I20260925 10:40:36.194275 2317 ts_tablet_manager.cc:1939] T 1b068e782424424287d236c2a539568b P 98da1065428f4d2785dbd553a4ee7fc7: Deleting consensus metadata
I20260925 10:40:36.195729 1805 catalog_manager.cc:4841] TS 98da1065428f4d2785dbd553a4ee7fc7 (127.23.100.2:37303): tablet 1b068e782424424287d236c2a539568b (table test-table [id=384e0ea1d4c34fddba45a9bb6b2b1c43]) successfully deleted
I20260925 10:40:36.308072 1803 catalog_manager.cc:4841] TS 675bff02c9c9476bacaaa4446d01c5a0 (127.23.100.1:44967): tablet 1b068e782424424287d236c2a539568b (table test-table [id=384e0ea1d4c34fddba45a9bb6b2b1c43]) successfully deleted
I20260925 10:40:36.308454 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 1880
I20260925 10:40:36.331382 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 2014
Sep 25 10:40:36 dist-test-slave-dxdd krb5kdc[1774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.2@KRBTEST.COM
I20260925 10:40:36.346115 2288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:36.332200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37750 (local address 127.23.100.2:37303)
0925 10:40:36.332383 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:36.332391 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:36.332411 (+ 20us) server_negotiation.cc:407] Connection header received
0925 10:40:36.332610 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:36.332615 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:36.332698 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:36.332797 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:36.333966 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:36.334739 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:36.335961 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:36.336366 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:36.336494 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:36.340547 (+ 4053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:40:36.340565 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:40:36.340580 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:40:36.340614 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:40:36.343597 (+ 2983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:40:36.344204 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:40:36.344218 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:40:36.344220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:40:36.344336 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:40:36.344672 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:40:36.344678 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:40:36.344681 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:40:36.345024 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:40:36.345326 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:36.345572 (+ 246us) server_negotiation.cc:299] Negotiation successful
0925 10:40:36.345800 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 10:40:36.346732 2137 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:40:36.469324 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 2014
I20260925 10:40:36.485832 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 2147
Sep 25 10:40:36 dist-test-slave-dxdd krb5kdc[1774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.3@KRBTEST.COM
I20260925 10:40:36.501606 2292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:40:36.486587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60510 (local address 127.23.100.3:36099)
0925 10:40:36.486770 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 10:40:36.486780 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:40:36.486882 (+ 102us) server_negotiation.cc:407] Connection header received
0925 10:40:36.487083 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:40:36.487089 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:40:36.487163 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:40:36.487269 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:40:36.488718 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:36.489584 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:36.490863 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:40:36.491305 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:40:36.491469 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:40:36.495604 (+ 4135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:40:36.495619 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:40:36.495639 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:40:36.495686 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:40:36.498898 (+ 3212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:40:36.499527 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:40:36.499536 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:40:36.499540 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:40:36.499691 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:40:36.500103 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:40:36.500109 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:40:36.500112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:40:36.500455 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:40:36.500763 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0925 10:40:36.501032 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 10:40:36.501238 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 10:40:36.502224 2270 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:40:36.626537 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 2147
I20260925 10:40:36.646631 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 1787
I20260925 10:40:36.647722 1849 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:40:36.753584 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 1787
2026-09-25T10:40:36Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4345 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:40:36 dist-test-slave-dxdd krb5kdc[2326](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:40:36 dist-test-slave-dxdd krb5kdc[2326](info): set up 2 sockets
Sep 25 10:40:36 dist-test-slave-dxdd krb5kdc[2326](info): commencing operation
krb5kdc: starting...
W20260925 10:40:37.847400 23952 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.005s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Sep 25 10:40:37 dist-test-slave-dxdd krb5kdc[2326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:40:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:40:38Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.23.100.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.23.100.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.23.100.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/krb5kdc/rangeradmin_127.23.100.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.23.100.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/krb5kdc/rangeradmin_127.23.100.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.23.100.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.23.100.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.23.100.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/krb5kdc/rangerlookup_127.23.100.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.23.100.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/krb5kdc/rangerlookup_127.23.100.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.100.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.100.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.100.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/krb5kdc/HTTP_127.23.100.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.100.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/krb5kdc/HTTP_127.23.100.20@KRBTEST.COM.keytab.
I20260925 10:40:38.153107 23952 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskVr7Mel/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/postgres -l logfile start
2026-09-25 10:40:40.460 UTC [2356] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 10:40:40.460 UTC [2356] LOG: listening on IPv4 address "127.23.100.20", port 42849
2026-09-25 10:40:40.463 UTC [2356] LOG: listening on Unix socket "/tmp/.s.PGSQL.42849"
2026-09-25 10:40:40.485 UTC [2359] LOG: database system was shut down at 2026-09-25 10:40:39 UTC
2026-09-25 10:40:40.488 UTC [2356] LOG: database system is ready to accept connections
I20260925 10:40:41.455020 23952 mini_postgres.cc:96] Postgres bound to 42849
2026-09-25 10:40:41.472 UTC [2367] FATAL: database "slave" does not exist
127.23.100.20:42849 - accepting connections
I20260925 10:40:41.473219 23952 mini_ranger.cc:161] Starting Ranger...
I20260925 10:40:41.499053 23952 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 10:40:41.933753 23952 mini_ranger.cc:87] Created ranger Postgres database
I20260925 10:40:41.933861 23952 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ranger-admin
2026-09-25 10:40:42,224 [I] DB FLAVOR :POSTGRES
2026-09-25 10:40:42,225 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 10:40:42,225 [I] Checking connection..
2026-09-25 10:40:42,225 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVr7Mel/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVr7Mel/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.20:42849/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 10:40:42,470 [I] Checking connection passed.
2026-09-25 10:40:42,471 [I] --------- Verifying version history table ---------
2026-09-25 10:40:42,471 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVr7Mel/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVr7Mel/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.20:42849/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 10:40:42,716 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 10:40:42,716 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVr7Mel/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVr7Mel/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.20:42849/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 10:40:42,963 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 10:40:42,963 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 10:40:42,963 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVr7Mel/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVr7Mel/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.20:42849/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskVr7Mel/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 10:40:43.212 UTC [2431] WARNING: there is no transaction in progress
2026-09-25 10:40:43,217 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 10:40:43,218 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVr7Mel/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVr7Mel/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.20:42849/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 10:40:43,466 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 10:40:43,466 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 10:40:43,467 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVr7Mel/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVr7Mel/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.20:42849/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 10:40:43,692 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVr7Mel/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVr7Mel/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.20:42849/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 10:40:43,923 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVr7Mel/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVr7Mel/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.20:42849/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-dxdd.c.gcp-upstream.internal','N') ;"
2026-09-25 10:40:44,147 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVr7Mel/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVr7Mel/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.20:42849/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 10:40:44,400 [I] Table x_portal_user does not exist in database ranger
2026-09-25 10:40:44,400 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVr7Mel/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVr7Mel/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.20:42849/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 10:40:44,621 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 10:40:44,622 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVr7Mel/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVr7Mel/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.20:42849/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 10:40:44,874 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVr7Mel/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVr7Mel/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.20:42849/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 10:40:45,128 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 10:40:45,128 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVr7Mel/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVr7Mel/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.20:42849/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskVr7Mel/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 10:40:46.058 UTC [2539] WARNING: there is no transaction in progress
2026-09-25 10:40:46.074 UTC [2539] WARNING: there is no transaction in progress
2026-09-25 10:40:46.088 UTC [2539] WARNING: there is no transaction in progress
2026-09-25 10:40:46.102 UTC [2539] WARNING: there is no transaction in progress
2026-09-25 10:40:46.117 UTC [2539] WARNING: there is no transaction in progress
2026-09-25 10:40:46.133 UTC [2539] WARNING: there is no transaction in progress
2026-09-25 10:40:46.184 UTC [2539] WARNING: there is no transaction in progress
2026-09-25 10:40:46.190 UTC [2539] WARNING: there is no transaction in progress
2026-09-25 10:40:46.203 UTC [2539] WARNING: there is no transaction in progress
2026-09-25 10:40:46.213 UTC [2539] WARNING: there is no transaction in progress
2026-09-25 10:40:46.222 UTC [2539] WARNING: there is no transaction in progress
2026-09-25 10:40:46.234 UTC [2539] WARNING: there is no transaction in progress
2026-09-25 10:40:46.242 UTC [2539] WARNING: there is no transaction in progress
2026-09-25 10:40:46.248 UTC [2539] WARNING: there is no transaction in progress
2026-09-25 10:40:46.254 UTC [2539] WARNING: there is no transaction in progress
2026-09-25 10:40:46.259 UTC [2539] WARNING: there is no transaction in progress
2026-09-25 10:40:46,846 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 10:40:46,846 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVr7Mel/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVr7Mel/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.20:42849/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 10:40:47,092 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 10:40:47,093 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVr7Mel/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVr7Mel/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.20:42849/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 10:40:47,347 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 10:40:47,347 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVr7Mel/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVr7Mel/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.20:42849/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 10:40:47,574 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 10:40:47,574 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVr7Mel/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVr7Mel/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.20:42849/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 10:40:47,825 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVr7Mel/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVr7Mel/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.20:42849/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 10:40:48,069 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVr7Mel/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVr7Mel/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.20:42849/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-dxdd.c.gcp-upstream.internal';"
2026-09-25 10:40:48,300 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 10:40:48,300 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVr7Mel/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVr7Mel/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.20:42849/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 10:40:48,536 [I] DB_PATCHES have already been applied
I20260925 10:40:48.542021 23952 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ranger-admin:/tmp/dist-test-taskVr7Mel/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskVr7Mel/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskVr7Mel/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVr7Mel/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 10:40:48.542105 23952 mini_ranger.cc:193] Using host: 127.23.100.20
I20260925 10:40:48.556613 23952 mini_ranger.cc:227] Ranger admin URL: http://127.23.100.20:41337
Sep 25, 2026 10:40:48 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 10:40:48 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 10:40:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 10:40:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 10:40:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 10:40:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 10:40:49 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 10:40:49.668627 23952 mini_ranger.cc:160] Time spent starting Ranger: real 8.195s user 0.007s sys 0.032s
Sep 25, 2026 10:41:09 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 10:41:09 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 25, 2026 10:41:09 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 10:41:09 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 10:41:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 10:41:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 10:41:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 10:41:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 10:41:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 10:41:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 10:41:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 10:41:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 10:41:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 10:41:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 10:41:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 10:41:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 10:41:09 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 10:41:10 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 10:41:10.864174 23952 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.23.100.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.23.100.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.23.100.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/krb5kdc/rangerkms_127.23.100.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.23.100.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/krb5kdc/rangerkms_127.23.100.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.100.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.23.100.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.100.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/krb5kdc/HTTP_127.23.100.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.100.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/krb5kdc/HTTP_127.23.100.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 10:41:11 dist-test-slave-dxdd krb5kdc[2326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332871, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 10:41:11.015512 23952 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 10:41:11.033947 23952 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 10:41:11.862718 23952 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 10:41:11.862897 23952 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ranger-kms
I20260925 10:41:11.863008 23952 mini_ranger_kms.cc:227] Using postgres at 127.23.100.20:42849
2026-09-25 10:41:12,271 [I] DB FLAVOR :POSTGRES
2026-09-25 10:41:12,271 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 10:41:12,271 [I] Checking connection
2026-09-25 10:41:12,271 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVr7Mel/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.20:42849/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 10:41:12,517 [I] connection success
2026-09-25 10:41:12,518 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 10:41:12,518 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 10:41:12,518 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVr7Mel/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.20:42849/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 10:41:12,750 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 10:41:12,750 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 10:41:12,750 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 10:41:12,750 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVr7Mel/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.20:42849/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 10:41:13,006 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 10:41:13.450553 23952 mini_ranger_kms.cc:310] Created kms service
I20260925 10:41:13.618825 23952 mini_ranger_kms.cc:326] Created kudu user
I20260925 10:41:13.659468 23952 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 10:41:13 dist-test-slave-dxdd krb5kdc[2326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332871, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.23.100.20@KRBTEST.COM
I20260925 10:41:14.040287 23952 mini_ranger_kms.cc:384] Added ranger policy
I20260925 10:41:14.040594 23952 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ranger-kms:/tmp/dist-test-taskVr7Mel/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVr7Mel/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskVr7Mel/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 10:41:14.040649 23952 mini_ranger_kms.cc:240] Using host: 127.23.100.20
I20260925 10:41:14.065191 23952 mini_ranger_kms.cc:276] Ranger KMS PID: 2719
I20260925 10:41:14.065377 23952 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.23.100.20:57805
Sep 25, 2026 10:41:15 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 10:41:15 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 10:41:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 10:41:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 10:41:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 10:41:15 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 10:41:16.347370 23952 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.332s user 0.117s sys 0.309s
I20260925 10:41:16.347628 23952 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 10:41:16.347852 23952 mini_ranger_kms.cc:401] 127.23.100.20:57805/kms/v1/keys
Sep 25 10:41:16 dist-test-slave-dxdd krb5kdc[2326](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790332876, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.100.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 10:41:16 dist-test-slave-dxdd krb5kdc[2326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332876, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.100.20@KRBTEST.COM for HTTP/127.23.100.20@KRBTEST.COM
Sep 25 10:41:17 dist-test-slave-dxdd krb5kdc[2326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332876, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.100.20@KRBTEST.COM for HTTP/127.23.100.20@KRBTEST.COM
Sep 25 10:41:19 dist-test-slave-dxdd krb5kdc[2326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.23.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
I20260925 10:41:19.445724 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:46685
--webserver_interface=127.23.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:33905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.62:46685
--ranger_config_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.100.20:57805/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:41:19.769622 2770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:41:19.769949 2770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:41:19.770109 2770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:41:19.777699 2770 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:41:19.777833 2770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:41:19.777931 2770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:41:19.778033 2770 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:41:19.778105 2770 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:41:19.787585 2770 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:33905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/master-0/wal
--ranger_kms_url=127.23.100.20:57805/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.23.100.62:46685
--ranger_config_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:46685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:41:19.788803 2770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:41:19.790038 2770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:41:19.801498 2775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:41:19.801829 2770 server_base.cc:1034] running on GCE node
W20260925 10:41:19.801584 2778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:41:19.801591 2776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:41:19.802877 2770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:41:19.804162 2770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:41:19.805390 2770 hybrid_clock.cc:648] HybridClock initialized: now 1790332879805341 us; error 43 us; skew 500 ppm
I20260925 10:41:19.805724 2770 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:41:19 dist-test-slave-dxdd krb5kdc[2326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332879, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:41:19.808810 2770 init.cc:377] Logged in from keytab as kudu/127.23.100.62@KRBTEST.COM (short username kudu)
I20260925 10:41:19.810129 2770 webserver.cc:460] Webserver started at http://127.23.100.62:38811/ using document root <none> and password file <none>
I20260925 10:41:19.810779 2770 fs_manager.cc:359] Metadata directory not provided
I20260925 10:41:19.810916 2770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:41:19.811210 2770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 10:41:19 dist-test-slave-dxdd krb5kdc[2326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332879, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.62@KRBTEST.COM for HTTP/127.23.100.20@KRBTEST.COM
I20260925 10:41:19.868845 2770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/master-0/data/instance:
uuid: "ca9a142d17bf4079a897f2db9419c51c"
format_stamp: "Formatted at 2026-09-25 10:41:19 on dist-test-slave-dxdd"
server_key: "66c2b4d60ec01006caff6df7c02fd30f"
server_key_iv: "13de9cc6a760ba180867d5ef853e179b"
server_key_version: "kuduclusterkey@0"
I20260925 10:41:19.869872 2770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/master-0/wal/instance:
uuid: "ca9a142d17bf4079a897f2db9419c51c"
format_stamp: "Formatted at 2026-09-25 10:41:19 on dist-test-slave-dxdd"
server_key: "66c2b4d60ec01006caff6df7c02fd30f"
server_key_iv: "13de9cc6a760ba180867d5ef853e179b"
server_key_version: "kuduclusterkey@0"
I20260925 10:41:19.875758 2770 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260925 10:41:19.892834 2785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:41:19.894407 2770 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:41:19.894567 2770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/master-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/master-0/wal
uuid: "ca9a142d17bf4079a897f2db9419c51c"
format_stamp: "Formatted at 2026-09-25 10:41:19 on dist-test-slave-dxdd"
server_key: "66c2b4d60ec01006caff6df7c02fd30f"
server_key_iv: "13de9cc6a760ba180867d5ef853e179b"
server_key_version: "kuduclusterkey@0"
I20260925 10:41:19.894739 2770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:41:19.906368 2770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:41:19.909646 2770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:41:19.909894 2770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:41:19.928144 2770 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.62:46685
I20260925 10:41:19.928176 2836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.62:46685 every 8 connection(s)
I20260925 10:41:19.929374 2770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
I20260925 10:41:19.932955 2837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:41:19.933266 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 2770
I20260925 10:41:19.933478 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/master-0/wal/instance
I20260925 10:41:19.934271 23952 external_mini_cluster.cc:1428] Setting key 4ce89efc24ea3a2ce0d547ddea05f925
I20260925 10:41:19.946244 2837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca9a142d17bf4079a897f2db9419c51c: Bootstrap starting.
I20260925 10:41:19.949024 2837 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca9a142d17bf4079a897f2db9419c51c: Neither blocks nor log segments found. Creating new log.
I20260925 10:41:19.949923 2837 log.cc:824] T 00000000000000000000000000000000 P ca9a142d17bf4079a897f2db9419c51c: Log is configured to *not* fsync() on all Append() calls
I20260925 10:41:19.952587 2837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca9a142d17bf4079a897f2db9419c51c: No bootstrap required, opened a new log
Sep 25 10:41:19 dist-test-slave-dxdd krb5kdc[2326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:41:19.960230 2837 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca9a142d17bf4079a897f2db9419c51c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca9a142d17bf4079a897f2db9419c51c" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 46685 } }
I20260925 10:41:19.960562 2837 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca9a142d17bf4079a897f2db9419c51c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:41:19.960680 2837 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca9a142d17bf4079a897f2db9419c51c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca9a142d17bf4079a897f2db9419c51c, State: Initialized, Role: FOLLOWER
I20260925 10:41:19.961212 2837 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca9a142d17bf4079a897f2db9419c51c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca9a142d17bf4079a897f2db9419c51c" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 46685 } }
I20260925 10:41:19.961445 2837 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca9a142d17bf4079a897f2db9419c51c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:41:19.961608 2837 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca9a142d17bf4079a897f2db9419c51c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:41:19.961781 2837 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca9a142d17bf4079a897f2db9419c51c [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:41:19.962997 2840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:41:19.936255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48644 (local address 127.23.100.62:46685)
0925 10:41:19.936994 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0925 10:41:19.937005 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:41:19.937043 (+ 38us) server_negotiation.cc:407] Connection header received
0925 10:41:19.939067 (+ 2024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:41:19.939090 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:41:19.939478 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:41:19.939800 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:41:19.941267 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:41:19.942577 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:41:19.944359 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:41:19.944982 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:41:19.945099 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:41:19.955535 (+ 10436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:41:19.955542 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:41:19.955555 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:41:19.955587 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:41:19.958652 (+ 3065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:41:19.959331 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:41:19.959337 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:41:19.959342 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:41:19.959470 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:41:19.959846 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:41:19.959853 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:41:19.959856 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:41:19.960358 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:41:19.960635 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0925 10:41:19.961632 (+ 997us) server_negotiation.cc:299] Negotiation successful
0925 10:41:19.962013 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":343,"threads_started":1}
I20260925 10:41:19.964085 2837 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca9a142d17bf4079a897f2db9419c51c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca9a142d17bf4079a897f2db9419c51c" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 46685 } }
I20260925 10:41:19.964602 2837 leader_election.cc:302] T 00000000000000000000000000000000 P ca9a142d17bf4079a897f2db9419c51c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca9a142d17bf4079a897f2db9419c51c; no voters:
I20260925 10:41:19.965152 2837 leader_election.cc:288] T 00000000000000000000000000000000 P ca9a142d17bf4079a897f2db9419c51c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:41:19.965274 2842 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca9a142d17bf4079a897f2db9419c51c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:41:19.966267 2842 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca9a142d17bf4079a897f2db9419c51c [term 1 LEADER]: Becoming Leader. State: Replica: ca9a142d17bf4079a897f2db9419c51c, State: Running, Role: LEADER
I20260925 10:41:19.966796 2842 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca9a142d17bf4079a897f2db9419c51c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca9a142d17bf4079a897f2db9419c51c" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 46685 } }
I20260925 10:41:19.967140 2837 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca9a142d17bf4079a897f2db9419c51c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:41:19.969332 2843 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca9a142d17bf4079a897f2db9419c51c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca9a142d17bf4079a897f2db9419c51c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca9a142d17bf4079a897f2db9419c51c" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 46685 } } }
I20260925 10:41:19.969936 2843 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca9a142d17bf4079a897f2db9419c51c [sys.catalog]: This master's current role is: LEADER
I20260925 10:41:19.969832 2844 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca9a142d17bf4079a897f2db9419c51c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca9a142d17bf4079a897f2db9419c51c. Latest consensus state: current_term: 1 leader_uuid: "ca9a142d17bf4079a897f2db9419c51c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca9a142d17bf4079a897f2db9419c51c" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 46685 } } }
I20260925 10:41:19.974628 2844 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca9a142d17bf4079a897f2db9419c51c [sys.catalog]: This master's current role is: LEADER
I20260925 10:41:19.976065 2851 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:41:19.980516 2851 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:41:19.981240 2837 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 10:41:20.000619 2851 catalog_manager.cc:1347] Generated new cluster ID: 484defcdfca442368d71d3be1654b6b0
I20260925 10:41:20.000833 2851 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:41:20.023211 2851 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:41:20.025127 2851 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:41:20.036170 2851 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca9a142d17bf4079a897f2db9419c51c: Generated new TSK 0
I20260925 10:41:20.037415 2851 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab.
Entry for principal kudu/127.23.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab.
I20260925 10:41:20.732478 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.100.20:57805/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.1:0
--local_ip_for_outbound_sockets=127.23.100.1
--webserver_interface=127.23.100.1
--webserver_port=0
--tserver_master_addrs=127.23.100.62:46685
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab
--principal=kudu/127.23.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:33905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 10:41:20 dist-test-slave-dxdd krb5kdc[2326](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790332880, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 10:41:21.150091 2882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:41:21.150522 2882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:41:21.150698 2882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:41:21.162330 2882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:41:21.162467 2882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:41:21.162647 2882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.1
I20260925 10:41:21.178213 2882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:33905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.23.100.20:57805/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab
--principal=kudu/127.23.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.1
--webserver_port=0
--tserver_master_addrs=127.23.100.62:46685
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.1
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:41:21.180234 2882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:41:21.182022 2882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:41:21.199460 2892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:41:21.199450 2893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:41:21.206184 2882 server_base.cc:1034] running on GCE node
W20260925 10:41:21.206940 2896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:41:21.207715 2882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:41:21.208909 2882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:41:21.210192 2882 hybrid_clock.cc:648] HybridClock initialized: now 1790332881210126 us; error 49 us; skew 500 ppm
I20260925 10:41:21.210594 2882 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:41:21 dist-test-slave-dxdd krb5kdc[2326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332881, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:41:21.213789 2882 init.cc:377] Logged in from keytab as kudu/127.23.100.1@KRBTEST.COM (short username kudu)
I20260925 10:41:21.215488 2882 webserver.cc:460] Webserver started at http://127.23.100.1:33505/ using document root <none> and password file <none>
I20260925 10:41:21.216164 2882 fs_manager.cc:359] Metadata directory not provided
I20260925 10:41:21.216289 2882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:41:21.216603 2882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 10:41:21 dist-test-slave-dxdd krb5kdc[2326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332881, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.1@KRBTEST.COM for HTTP/127.23.100.20@KRBTEST.COM
I20260925 10:41:21.235152 2882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ts-0/data/instance:
uuid: "895c5b9da05444b28c3b3f9d31a20acd"
format_stamp: "Formatted at 2026-09-25 10:41:21 on dist-test-slave-dxdd"
server_key: "93674d368826aaad50ccaac2ea08ef7c"
server_key_iv: "01412329a8ad11df7c8bfe23c3a61194"
server_key_version: "kuduclusterkey@0"
I20260925 10:41:21.236218 2882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ts-0/wal/instance:
uuid: "895c5b9da05444b28c3b3f9d31a20acd"
format_stamp: "Formatted at 2026-09-25 10:41:21 on dist-test-slave-dxdd"
server_key: "93674d368826aaad50ccaac2ea08ef7c"
server_key_iv: "01412329a8ad11df7c8bfe23c3a61194"
server_key_version: "kuduclusterkey@0"
I20260925 10:41:21.252467 2882 fs_manager.cc:691] Time spent creating directory manager: real 0.016s user 0.006s sys 0.002s
I20260925 10:41:21.283102 2902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:41:21.285197 2882 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.010s sys 0.000s
I20260925 10:41:21.285377 2882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ts-0/wal
uuid: "895c5b9da05444b28c3b3f9d31a20acd"
format_stamp: "Formatted at 2026-09-25 10:41:21 on dist-test-slave-dxdd"
server_key: "93674d368826aaad50ccaac2ea08ef7c"
server_key_iv: "01412329a8ad11df7c8bfe23c3a61194"
server_key_version: "kuduclusterkey@0"
I20260925 10:41:21.285630 2882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:41:21.341454 2882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:41:21.346591 2882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:41:21.346963 2882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:41:21.354291 2882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:41:21.362138 2882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:41:21.362260 2882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:41:21.362447 2882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:41:21.362532 2882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:41:21.423853 2882 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.1:36919
I20260925 10:41:21.423884 3014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.1:36919 every 8 connection(s)
I20260925 10:41:21.425642 2882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
I20260925 10:41:21.430181 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 2882
I20260925 10:41:21.430438 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ts-0/wal/instance
I20260925 10:41:21.431409 23952 external_mini_cluster.cc:1428] Setting key b94d671ca20c80877ae680e8c022c556
WARNING: no policy specified for kudu/127.23.100.2@KRBTEST.COM; defaulting to no policy
Sep 25 10:41:21 dist-test-slave-dxdd krb5kdc[2326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332881, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.1@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.2@KRBTEST.COM" created.
Sep 25 10:41:21 dist-test-slave-dxdd krb5kdc[2326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332880, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.62@KRBTEST.COM for HTTP/127.23.100.20@KRBTEST.COM
I20260925 10:41:21.482590 3017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:41:21.440617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:46685 (local address 127.23.100.1:50343)
0925 10:41:21.441590 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0925 10:41:21.441650 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 10:41:21.444762 (+ 3112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:41:21.454271 (+ 9509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:41:21.454286 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:41:21.454958 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:41:21.455796 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:41:21.455821 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:41:21.459769 (+ 3948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:41:21.459775 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:41:21.461985 (+ 2210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:41:21.462000 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:41:21.463780 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:41:21.463788 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:41:21.464070 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:41:21.464769 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:41:21.464798 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:41:21.467477 (+ 2679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:41:21.477776 (+ 10299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:41:21.477787 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:41:21.477807 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:41:21.478430 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:41:21.478933 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:41:21.478938 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:41:21.478941 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:41:21.479197 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:41:21.479891 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:41:21.479900 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:41:21.480334 (+ 434us) client_negotiation.cc:769] Sending connection context
0925 10:41:21.480635 (+ 301us) client_negotiation.cc:240] Negotiation successful
0925 10:41:21.481192 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":608,"threads_started":1}
I20260925 10:41:21.484500 3018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:41:21.441200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:50343 (local address 127.23.100.62:46685)
0925 10:41:21.451437 (+ 10237us) server_negotiation.cc:206] Beginning negotiation
0925 10:41:21.451445 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:41:21.451469 (+ 24us) server_negotiation.cc:407] Connection header received
0925 10:41:21.451535 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:41:21.451544 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:41:21.451643 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:41:21.458165 (+ 6522us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:41:21.458385 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:41:21.459547 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:41:21.462179 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:41:21.462862 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:41:21.464457 (+ 1595us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:41:21.467716 (+ 3259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:41:21.467723 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:41:21.467727 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:41:21.467763 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:41:21.477571 (+ 9808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:41:21.478609 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:41:21.478618 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:41:21.478625 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:41:21.478795 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:41:21.479347 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:41:21.479352 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:41:21.479356 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:41:21.479735 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:41:21.483751 (+ 4016us) server_negotiation.cc:1035] Waiting for connection context
0925 10:41:21.483896 (+ 145us) server_negotiation.cc:299] Negotiation successful
0925 10:41:21.484130 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10092,"thread_start_us":9984,"threads_started":1}
I20260925 10:41:21.486140 3015 heartbeater.cc:347] Connected to a master server at 127.23.100.62:46685
I20260925 10:41:21.486495 3015 heartbeater.cc:464] Registering TS with master...
I20260925 10:41:21.487314 3015 heartbeater.cc:511] Master 127.23.100.62:46685 requested a full tablet report, sending...
I20260925 10:41:21.489779 2802 ts_manager.cc:194] Registered new tserver with Master: 895c5b9da05444b28c3b3f9d31a20acd (127.23.100.1:36919)
I20260925 10:41:21.492157 2802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.1@KRBTEST.COM'} at 127.23.100.1:50343
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab.
Entry for principal kudu/127.23.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab.
I20260925 10:41:21.504637 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.100.20:57805/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.2:0
--local_ip_for_outbound_sockets=127.23.100.2
--webserver_interface=127.23.100.2
--webserver_port=0
--tserver_master_addrs=127.23.100.62:46685
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab
--principal=kudu/127.23.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:33905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 10:41:21 dist-test-slave-dxdd krb5kdc[2326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332880, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.62@KRBTEST.COM for HTTP/127.23.100.20@KRBTEST.COM
W20260925 10:41:21.990203 3021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:41:21.990712 3021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:41:21.990895 3021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:41:22.002841 3021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:41:22.003038 3021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:41:22.003273 3021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.2
I20260925 10:41:22.018965 3021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:33905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.23.100.20:57805/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab
--principal=kudu/127.23.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.2
--webserver_port=0
--tserver_master_addrs=127.23.100.62:46685
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.2
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:41:22.020999 3021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:41:22.022816 3021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:41:22.043560 3030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:41:22.044078 3031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:41:22.045944 3037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:41:22.047111 3021 server_base.cc:1034] running on GCE node
I20260925 10:41:22.047710 3021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:41:22.048714 3021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:41:22.049971 3021 hybrid_clock.cc:648] HybridClock initialized: now 1790332882049911 us; error 54 us; skew 500 ppm
I20260925 10:41:22.050352 3021 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:41:22 dist-test-slave-dxdd krb5kdc[2326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332882, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:41:22.053480 3021 init.cc:377] Logged in from keytab as kudu/127.23.100.2@KRBTEST.COM (short username kudu)
I20260925 10:41:22.055130 3021 webserver.cc:460] Webserver started at http://127.23.100.2:34299/ using document root <none> and password file <none>
I20260925 10:41:22.055786 3021 fs_manager.cc:359] Metadata directory not provided
I20260925 10:41:22.055904 3021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:41:22.056216 3021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 10:41:22 dist-test-slave-dxdd krb5kdc[2326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332882, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.2@KRBTEST.COM for HTTP/127.23.100.20@KRBTEST.COM
I20260925 10:41:22.072141 3021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ts-1/data/instance:
uuid: "c706c542cdf3492e95c148b302eb0ce8"
format_stamp: "Formatted at 2026-09-25 10:41:22 on dist-test-slave-dxdd"
server_key: "a9a416b1a9796ee0397a0ef979409f7c"
server_key_iv: "b31fb7f33a19ea84acce6c40532cd0ad"
server_key_version: "kuduclusterkey@0"
I20260925 10:41:22.072963 3021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ts-1/wal/instance:
uuid: "c706c542cdf3492e95c148b302eb0ce8"
format_stamp: "Formatted at 2026-09-25 10:41:22 on dist-test-slave-dxdd"
server_key: "a9a416b1a9796ee0397a0ef979409f7c"
server_key_iv: "b31fb7f33a19ea84acce6c40532cd0ad"
server_key_version: "kuduclusterkey@0"
I20260925 10:41:22.078219 3021 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260925 10:41:22.094890 3045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:41:22.096745 3021 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260925 10:41:22.096912 3021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ts-1/wal
uuid: "c706c542cdf3492e95c148b302eb0ce8"
format_stamp: "Formatted at 2026-09-25 10:41:22 on dist-test-slave-dxdd"
server_key: "a9a416b1a9796ee0397a0ef979409f7c"
server_key_iv: "b31fb7f33a19ea84acce6c40532cd0ad"
server_key_version: "kuduclusterkey@0"
I20260925 10:41:22.097153 3021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:41:22.119239 3021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:41:22.123219 3021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:41:22.123513 3021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:41:22.124495 3021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:41:22.125811 3021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:41:22.125913 3021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:41:22.126065 3021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:41:22.126138 3021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:41:22.169013 3021 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.2:34733
I20260925 10:41:22.169016 3157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.2:34733 every 8 connection(s)
I20260925 10:41:22.170382 3021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
I20260925 10:41:22.176245 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 3021
I20260925 10:41:22.176424 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ts-1/wal/instance
I20260925 10:41:22.176918 23952 external_mini_cluster.cc:1428] Setting key 838e3c9b835344ca135024d3536ab556
Sep 25 10:41:22 dist-test-slave-dxdd krb5kdc[2326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332882, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.2@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:41:22.191785 3160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:41:22.173287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:58307 (local address 127.23.100.62:46685)
0925 10:41:22.173805 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0925 10:41:22.173809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:41:22.174852 (+ 1043us) server_negotiation.cc:407] Connection header received
0925 10:41:22.176837 (+ 1985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:41:22.176843 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:41:22.176946 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:41:22.177065 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:41:22.179086 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:41:22.179810 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:41:22.181371 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:41:22.181781 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:41:22.181909 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:41:22.185230 (+ 3321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:41:22.185235 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:41:22.185239 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:41:22.185269 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:41:22.188067 (+ 2798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:41:22.188955 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:41:22.188963 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:41:22.188969 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:41:22.189121 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:41:22.189624 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:41:22.189632 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:41:22.189638 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:41:22.190026 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:41:22.190182 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 10:41:22.191021 (+ 839us) server_negotiation.cc:299] Negotiation successful
0925 10:41:22.191291 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":268,"threads_started":1}
I20260925 10:41:22.192724 3161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:41:22.173877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:46685 (local address 127.23.100.2:58307)
0925 10:41:22.174698 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0925 10:41:22.174735 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 10:41:22.176578 (+ 1843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:41:22.177319 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:41:22.177327 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:41:22.177855 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:41:22.178669 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:41:22.178695 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:41:22.179962 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:41:22.179968 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:41:22.181225 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:41:22.181236 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:41:22.182004 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:41:22.182011 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:41:22.182235 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:41:22.182967 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:41:22.182990 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:41:22.185005 (+ 2015us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:41:22.188215 (+ 3210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:41:22.188224 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:41:22.188242 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:41:22.188813 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:41:22.189238 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:41:22.189243 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:41:22.189247 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:41:22.189491 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:41:22.190173 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:41:22.190181 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:41:22.190578 (+ 397us) client_negotiation.cc:769] Sending connection context
0925 10:41:22.190859 (+ 281us) client_negotiation.cc:240] Negotiation successful
0925 10:41:22.191451 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":383,"threads_started":1}
I20260925 10:41:22.194233 3158 heartbeater.cc:347] Connected to a master server at 127.23.100.62:46685
I20260925 10:41:22.194576 3158 heartbeater.cc:464] Registering TS with master...
I20260925 10:41:22.195444 3158 heartbeater.cc:511] Master 127.23.100.62:46685 requested a full tablet report, sending...
I20260925 10:41:22.197098 2802 ts_manager.cc:194] Registered new tserver with Master: c706c542cdf3492e95c148b302eb0ce8 (127.23.100.2:34733)
I20260925 10:41:22.198228 2802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.2@KRBTEST.COM'} at 127.23.100.2:58307
WARNING: no policy specified for kudu/127.23.100.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab.
Entry for principal kudu/127.23.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab.
I20260925 10:41:22.234805 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.100.20:57805/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.3:0
--local_ip_for_outbound_sockets=127.23.100.3
--webserver_interface=127.23.100.3
--webserver_port=0
--tserver_master_addrs=127.23.100.62:46685
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab
--principal=kudu/127.23.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:33905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:41:22.495741 3015 heartbeater.cc:503] Master 127.23.100.62:46685 was elected leader, sending a full tablet report...
W20260925 10:41:22.550135 3164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:41:22.550433 3164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:41:22.550531 3164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:41:22.557379 3164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:41:22.557474 3164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:41:22.557577 3164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.3
I20260925 10:41:22.566840 3164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:33905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.23.100.20:57805/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab
--principal=kudu/127.23.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.3
--webserver_port=0
--tserver_master_addrs=127.23.100.62:46685
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.3
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:41:22.567996 3164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:41:22.569288 3164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:41:22.581821 3170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:41:22.581888 3169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:41:22.581916 3172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:41:22.582768 3164 server_base.cc:1034] running on GCE node
I20260925 10:41:22.583333 3164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:41:22.584219 3164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:41:22.585409 3164 hybrid_clock.cc:648] HybridClock initialized: now 1790332882585362 us; error 39 us; skew 500 ppm
I20260925 10:41:22.585744 3164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:41:22 dist-test-slave-dxdd krb5kdc[2326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332882, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:41:22.588794 3164 init.cc:377] Logged in from keytab as kudu/127.23.100.3@KRBTEST.COM (short username kudu)
I20260925 10:41:22.590215 3164 webserver.cc:460] Webserver started at http://127.23.100.3:41283/ using document root <none> and password file <none>
I20260925 10:41:22.590930 3164 fs_manager.cc:359] Metadata directory not provided
I20260925 10:41:22.591073 3164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:41:22.591462 3164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 10:41:22 dist-test-slave-dxdd krb5kdc[2326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332882, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.3@KRBTEST.COM for HTTP/127.23.100.20@KRBTEST.COM
I20260925 10:41:22.606545 3164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ts-2/data/instance:
uuid: "7489582e8a984290b3e9e68d847ef481"
format_stamp: "Formatted at 2026-09-25 10:41:22 on dist-test-slave-dxdd"
server_key: "b8c191b5147019d66babf0b886ea12e3"
server_key_iv: "2d77be532d862c75fbf5a537257e083b"
server_key_version: "kuduclusterkey@0"
I20260925 10:41:22.607407 3164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ts-2/wal/instance:
uuid: "7489582e8a984290b3e9e68d847ef481"
format_stamp: "Formatted at 2026-09-25 10:41:22 on dist-test-slave-dxdd"
server_key: "b8c191b5147019d66babf0b886ea12e3"
server_key_iv: "2d77be532d862c75fbf5a537257e083b"
server_key_version: "kuduclusterkey@0"
I20260925 10:41:22.612792 3164 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 10:41:22.628338 3179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:41:22.629658 3164 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:41:22.629853 3164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ts-2/wal
uuid: "7489582e8a984290b3e9e68d847ef481"
format_stamp: "Formatted at 2026-09-25 10:41:22 on dist-test-slave-dxdd"
server_key: "b8c191b5147019d66babf0b886ea12e3"
server_key_iv: "2d77be532d862c75fbf5a537257e083b"
server_key_version: "kuduclusterkey@0"
I20260925 10:41:22.630075 3164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:41:22.660439 3164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:41:22.664126 3164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:41:22.664419 3164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:41:22.665321 3164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:41:22.666772 3164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:41:22.666882 3164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:41:22.667047 3164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:41:22.667127 3164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:41:22.699043 3164 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.3:33251
I20260925 10:41:22.699085 3291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.3:33251 every 8 connection(s)
I20260925 10:41:22.700237 3164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
I20260925 10:41:22.707532 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 3164
I20260925 10:41:22.707711 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ts-2/wal/instance
I20260925 10:41:22.708137 23952 external_mini_cluster.cc:1428] Setting key 92ebbb9f3e5a33fc4181da92acc038c9
Sep 25 10:41:22 dist-test-slave-dxdd krb5kdc[2326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332882, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.3@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:41:22.718722 3294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:41:22.702640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:42135 (local address 127.23.100.62:46685)
0925 10:41:22.703129 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0925 10:41:22.703136 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:41:22.704168 (+ 1032us) server_negotiation.cc:407] Connection header received
0925 10:41:22.705857 (+ 1689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:41:22.705861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:41:22.705933 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:41:22.706029 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:41:22.707433 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:41:22.708238 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:41:22.710287 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:41:22.710843 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:41:22.710951 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:41:22.713894 (+ 2943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:41:22.713898 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:41:22.713902 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:41:22.713937 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:41:22.716311 (+ 2374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:41:22.716891 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:41:22.716894 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:41:22.716896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:41:22.717007 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:41:22.717350 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:41:22.717353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:41:22.717355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:41:22.717571 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:41:22.717669 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 10:41:22.718218 (+ 549us) server_negotiation.cc:299] Negotiation successful
0925 10:41:22.718422 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":203,"threads_started":1}
I20260925 10:41:22.719645 3295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:41:22.703223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:46685 (local address 127.23.100.3:42135)
0925 10:41:22.704014 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0925 10:41:22.704062 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 10:41:22.705630 (+ 1568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:41:22.706222 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:41:22.706230 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:41:22.706735 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:41:22.707274 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:41:22.707290 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:41:22.708389 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:41:22.708401 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:41:22.710154 (+ 1753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:41:22.710171 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:41:22.711060 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:41:22.711067 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:41:22.711262 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:41:22.711840 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:41:22.711858 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:41:22.713719 (+ 1861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:41:22.716419 (+ 2700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:41:22.716425 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:41:22.716435 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:41:22.716802 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:41:22.717091 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:41:22.717093 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:41:22.717096 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:41:22.717265 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:41:22.717670 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:41:22.717675 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:41:22.717950 (+ 275us) client_negotiation.cc:769] Sending connection context
0925 10:41:22.718148 (+ 198us) client_negotiation.cc:240] Negotiation successful
0925 10:41:22.718564 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":340,"threads_started":1}
I20260925 10:41:22.720840 3292 heartbeater.cc:347] Connected to a master server at 127.23.100.62:46685
I20260925 10:41:22.721092 3292 heartbeater.cc:464] Registering TS with master...
I20260925 10:41:22.721800 3292 heartbeater.cc:511] Master 127.23.100.62:46685 requested a full tablet report, sending...
I20260925 10:41:22.723145 2802 ts_manager.cc:194] Registered new tserver with Master: 7489582e8a984290b3e9e68d847ef481 (127.23.100.3:33251)
I20260925 10:41:22.723903 2802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.3@KRBTEST.COM'} at 127.23.100.3:42135
I20260925 10:41:22.735548 23952 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:41:22.751398 3294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:41:22.740317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54458 (local address 127.23.100.62:46685)
0925 10:41:22.740530 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 10:41:22.740539 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:41:22.740889 (+ 350us) server_negotiation.cc:407] Connection header received
0925 10:41:22.740992 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:41:22.740995 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:41:22.741071 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:41:22.741144 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:41:22.742458 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:41:22.743100 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:41:22.744393 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:41:22.744784 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:41:22.744870 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:41:22.747039 (+ 2169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:41:22.747044 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:41:22.747048 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:41:22.747075 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:41:22.749266 (+ 2191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:41:22.749813 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:41:22.749816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:41:22.749818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:41:22.749937 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:41:22.750228 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:41:22.750231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:41:22.750233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:41:22.750544 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:41:22.750702 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 10:41:22.750936 (+ 234us) server_negotiation.cc:299] Negotiation successful
0925 10:41:22.751077 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 10:41:22.754912 2802 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54458:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:41:22.762040 2802 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:41:22.795770 3305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:41:22.773524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:36919 (local address 127.0.0.1:32990)
0925 10:41:22.774228 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0925 10:41:22.774270 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 10:41:22.774481 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:41:22.788287 (+ 13806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:41:22.788295 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:41:22.788324 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:41:22.788860 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:41:22.788876 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:41:22.791015 (+ 2139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:41:22.791020 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:41:22.793006 (+ 1986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:41:22.793017 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:41:22.794766 (+ 1749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:41:22.794772 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:41:22.794975 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:41:22.795032 (+ 57us) client_negotiation.cc:769] Sending connection context
0925 10:41:22.795157 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 10:41:22.795317 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"mutex_wait_us":24,"thread_start_us":297,"threads_started":1}
I20260925 10:41:22.796557 3306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:41:22.773841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32990 (local address 127.23.100.1:36919)
0925 10:41:22.787448 (+ 13607us) server_negotiation.cc:206] Beginning negotiation
0925 10:41:22.787460 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:41:22.787496 (+ 36us) server_negotiation.cc:407] Connection header received
0925 10:41:22.787633 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:41:22.787644 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:41:22.787991 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:41:22.789052 (+ 1061us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:41:22.789280 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:41:22.790854 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:41:22.793152 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:41:22.794592 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:41:22.794710 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:41:22.794940 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0925 10:41:22.796218 (+ 1278us) server_negotiation.cc:299] Negotiation successful
0925 10:41:22.796341 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13394,"thread_start_us":13219,"threads_started":1}
I20260925 10:41:22.796614 3307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:41:22.774795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.2:34733 (local address 127.0.0.1:38714)
0925 10:41:22.788858 (+ 14063us) negotiation.cc:107] Waiting for socket to connect
0925 10:41:22.788874 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:41:22.788986 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:41:22.789988 (+ 1002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:41:22.789993 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:41:22.790017 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:41:22.790432 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:41:22.790442 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:41:22.792133 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:41:22.792138 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:41:22.793535 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:41:22.793547 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:41:22.795810 (+ 2263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:41:22.795820 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:41:22.795981 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:41:22.796010 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 10:41:22.796089 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 10:41:22.796209 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13916,"mutex_wait_us":1,"thread_start_us":13155,"threads_started":1}
I20260925 10:41:22.797091 3308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:41:22.774237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.3:33251 (local address 127.0.0.1:49886)
0925 10:41:22.788213 (+ 13976us) negotiation.cc:107] Waiting for socket to connect
0925 10:41:22.788230 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:41:22.788352 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:41:22.789063 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:41:22.789070 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:41:22.789096 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:41:22.789441 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:41:22.789449 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:41:22.791274 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:41:22.791279 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:41:22.794405 (+ 3126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:41:22.794420 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:41:22.796345 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:41:22.796351 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:41:22.796527 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:41:22.796567 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 10:41:22.796658 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 10:41:22.796787 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13854,"mutex_wait_us":32,"thread_start_us":14834,"threads_started":1}
I20260925 10:41:22.797291 3310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:41:22.775339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38714 (local address 127.23.100.2:34733)
0925 10:41:22.787718 (+ 12379us) server_negotiation.cc:206] Beginning negotiation
0925 10:41:22.787749 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0925 10:41:22.788979 (+ 1230us) server_negotiation.cc:407] Connection header received
0925 10:41:22.789635 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:41:22.789640 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:41:22.789811 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:41:22.789997 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:41:22.790569 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:41:22.791972 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:41:22.793666 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:41:22.795375 (+ 1709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:41:22.795491 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:41:22.795672 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 10:41:22.796803 (+ 1131us) server_negotiation.cc:299] Negotiation successful
0925 10:41:22.796965 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12204,"thread_start_us":12084,"threads_started":1}
I20260925 10:41:22.798226 3309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:41:22.774917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49886 (local address 127.23.100.3:33251)
0925 10:41:22.787064 (+ 12147us) server_negotiation.cc:206] Beginning negotiation
0925 10:41:22.787074 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:41:22.788314 (+ 1240us) server_negotiation.cc:407] Connection header received
0925 10:41:22.788480 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:41:22.788490 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:41:22.788726 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:41:22.788950 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:41:22.789618 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:41:22.791127 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:41:22.794582 (+ 3455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:41:22.795883 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:41:22.795992 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:41:22.796203 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 10:41:22.797291 (+ 1088us) server_negotiation.cc:299] Negotiation successful
0925 10:41:22.797445 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11958,"thread_start_us":11780,"threads_started":1}
I20260925 10:41:22.800908 3093 tablet_service.cc:1467] Processing CreateTablet for tablet 882624a8f1804980b7e6be4671b32c6b (DEFAULT_TABLE table=test-table [id=820c731390ff434190e0676520d2e856]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:41:22.800998 2950 tablet_service.cc:1467] Processing CreateTablet for tablet 882624a8f1804980b7e6be4671b32c6b (DEFAULT_TABLE table=test-table [id=820c731390ff434190e0676520d2e856]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:41:22.801328 3227 tablet_service.cc:1467] Processing CreateTablet for tablet 882624a8f1804980b7e6be4671b32c6b (DEFAULT_TABLE table=test-table [id=820c731390ff434190e0676520d2e856]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:41:22.802196 3093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 882624a8f1804980b7e6be4671b32c6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:41:22.802453 2950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 882624a8f1804980b7e6be4671b32c6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:41:22.802587 3227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 882624a8f1804980b7e6be4671b32c6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:41:22.813294 3311 tablet_bootstrap.cc:492] T 882624a8f1804980b7e6be4671b32c6b P c706c542cdf3492e95c148b302eb0ce8: Bootstrap starting.
I20260925 10:41:22.816344 3313 tablet_bootstrap.cc:492] T 882624a8f1804980b7e6be4671b32c6b P 7489582e8a984290b3e9e68d847ef481: Bootstrap starting.
I20260925 10:41:22.816619 3311 tablet_bootstrap.cc:654] T 882624a8f1804980b7e6be4671b32c6b P c706c542cdf3492e95c148b302eb0ce8: Neither blocks nor log segments found. Creating new log.
I20260925 10:41:22.817834 3312 tablet_bootstrap.cc:492] T 882624a8f1804980b7e6be4671b32c6b P 895c5b9da05444b28c3b3f9d31a20acd: Bootstrap starting.
I20260925 10:41:22.817900 3311 log.cc:824] T 882624a8f1804980b7e6be4671b32c6b P c706c542cdf3492e95c148b302eb0ce8: Log is configured to *not* fsync() on all Append() calls
I20260925 10:41:22.820713 3313 tablet_bootstrap.cc:654] T 882624a8f1804980b7e6be4671b32c6b P 7489582e8a984290b3e9e68d847ef481: Neither blocks nor log segments found. Creating new log.
I20260925 10:41:22.821077 3311 tablet_bootstrap.cc:492] T 882624a8f1804980b7e6be4671b32c6b P c706c542cdf3492e95c148b302eb0ce8: No bootstrap required, opened a new log
I20260925 10:41:22.821157 3312 tablet_bootstrap.cc:654] T 882624a8f1804980b7e6be4671b32c6b P 895c5b9da05444b28c3b3f9d31a20acd: Neither blocks nor log segments found. Creating new log.
I20260925 10:41:22.821336 3311 ts_tablet_manager.cc:1397] T 882624a8f1804980b7e6be4671b32c6b P c706c542cdf3492e95c148b302eb0ce8: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260925 10:41:22.822067 3313 log.cc:824] T 882624a8f1804980b7e6be4671b32c6b P 7489582e8a984290b3e9e68d847ef481: Log is configured to *not* fsync() on all Append() calls
I20260925 10:41:22.822342 3312 log.cc:824] T 882624a8f1804980b7e6be4671b32c6b P 895c5b9da05444b28c3b3f9d31a20acd: Log is configured to *not* fsync() on all Append() calls
I20260925 10:41:22.825013 3312 tablet_bootstrap.cc:492] T 882624a8f1804980b7e6be4671b32c6b P 895c5b9da05444b28c3b3f9d31a20acd: No bootstrap required, opened a new log
I20260925 10:41:22.825196 3313 tablet_bootstrap.cc:492] T 882624a8f1804980b7e6be4671b32c6b P 7489582e8a984290b3e9e68d847ef481: No bootstrap required, opened a new log
I20260925 10:41:22.825250 3312 ts_tablet_manager.cc:1397] T 882624a8f1804980b7e6be4671b32c6b P 895c5b9da05444b28c3b3f9d31a20acd: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260925 10:41:22.825405 3313 ts_tablet_manager.cc:1397] T 882624a8f1804980b7e6be4671b32c6b P 7489582e8a984290b3e9e68d847ef481: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260925 10:41:22.829373 3311 raft_consensus.cc:348] T 882624a8f1804980b7e6be4671b32c6b P c706c542cdf3492e95c148b302eb0ce8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895c5b9da05444b28c3b3f9d31a20acd" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 36919 } } peers { permanent_uuid: "c706c542cdf3492e95c148b302eb0ce8" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 34733 } } peers { permanent_uuid: "7489582e8a984290b3e9e68d847ef481" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 33251 } }
I20260925 10:41:22.829710 3311 raft_consensus.cc:374] T 882624a8f1804980b7e6be4671b32c6b P c706c542cdf3492e95c148b302eb0ce8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:41:22.829833 3311 raft_consensus.cc:729] T 882624a8f1804980b7e6be4671b32c6b P c706c542cdf3492e95c148b302eb0ce8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c706c542cdf3492e95c148b302eb0ce8, State: Initialized, Role: FOLLOWER
I20260925 10:41:22.830471 3311 consensus_queue.cc:260] T 882624a8f1804980b7e6be4671b32c6b P c706c542cdf3492e95c148b302eb0ce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895c5b9da05444b28c3b3f9d31a20acd" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 36919 } } peers { permanent_uuid: "c706c542cdf3492e95c148b302eb0ce8" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 34733 } } peers { permanent_uuid: "7489582e8a984290b3e9e68d847ef481" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 33251 } }
I20260925 10:41:22.831470 3158 heartbeater.cc:503] Master 127.23.100.62:46685 was elected leader, sending a full tablet report...
I20260925 10:41:22.832067 3311 ts_tablet_manager.cc:1428] T 882624a8f1804980b7e6be4671b32c6b P c706c542cdf3492e95c148b302eb0ce8: Time spent starting tablet: real 0.011s user 0.009s sys 0.003s
I20260925 10:41:22.835870 3313 raft_consensus.cc:348] T 882624a8f1804980b7e6be4671b32c6b P 7489582e8a984290b3e9e68d847ef481 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895c5b9da05444b28c3b3f9d31a20acd" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 36919 } } peers { permanent_uuid: "c706c542cdf3492e95c148b302eb0ce8" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 34733 } } peers { permanent_uuid: "7489582e8a984290b3e9e68d847ef481" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 33251 } }
I20260925 10:41:22.835870 3312 raft_consensus.cc:348] T 882624a8f1804980b7e6be4671b32c6b P 895c5b9da05444b28c3b3f9d31a20acd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895c5b9da05444b28c3b3f9d31a20acd" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 36919 } } peers { permanent_uuid: "c706c542cdf3492e95c148b302eb0ce8" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 34733 } } peers { permanent_uuid: "7489582e8a984290b3e9e68d847ef481" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 33251 } }
I20260925 10:41:22.836243 3313 raft_consensus.cc:374] T 882624a8f1804980b7e6be4671b32c6b P 7489582e8a984290b3e9e68d847ef481 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:41:22.836243 3312 raft_consensus.cc:374] T 882624a8f1804980b7e6be4671b32c6b P 895c5b9da05444b28c3b3f9d31a20acd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:41:22.836339 3313 raft_consensus.cc:729] T 882624a8f1804980b7e6be4671b32c6b P 7489582e8a984290b3e9e68d847ef481 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7489582e8a984290b3e9e68d847ef481, State: Initialized, Role: FOLLOWER
I20260925 10:41:22.836338 3312 raft_consensus.cc:729] T 882624a8f1804980b7e6be4671b32c6b P 895c5b9da05444b28c3b3f9d31a20acd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 895c5b9da05444b28c3b3f9d31a20acd, State: Initialized, Role: FOLLOWER
I20260925 10:41:22.836869 3313 consensus_queue.cc:260] T 882624a8f1804980b7e6be4671b32c6b P 7489582e8a984290b3e9e68d847ef481 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895c5b9da05444b28c3b3f9d31a20acd" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 36919 } } peers { permanent_uuid: "c706c542cdf3492e95c148b302eb0ce8" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 34733 } } peers { permanent_uuid: "7489582e8a984290b3e9e68d847ef481" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 33251 } }
I20260925 10:41:22.836867 3312 consensus_queue.cc:260] T 882624a8f1804980b7e6be4671b32c6b P 895c5b9da05444b28c3b3f9d31a20acd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895c5b9da05444b28c3b3f9d31a20acd" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 36919 } } peers { permanent_uuid: "c706c542cdf3492e95c148b302eb0ce8" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 34733 } } peers { permanent_uuid: "7489582e8a984290b3e9e68d847ef481" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 33251 } }
I20260925 10:41:22.837877 3292 heartbeater.cc:503] Master 127.23.100.62:46685 was elected leader, sending a full tablet report...
I20260925 10:41:22.838512 3313 ts_tablet_manager.cc:1428] T 882624a8f1804980b7e6be4671b32c6b P 7489582e8a984290b3e9e68d847ef481: Time spent starting tablet: real 0.013s user 0.015s sys 0.000s
I20260925 10:41:22.838574 3312 ts_tablet_manager.cc:1428] T 882624a8f1804980b7e6be4671b32c6b P 895c5b9da05444b28c3b3f9d31a20acd: Time spent starting tablet: real 0.013s user 0.012s sys 0.001s
I20260925 10:41:22.906672 3318 raft_consensus.cc:482] T 882624a8f1804980b7e6be4671b32c6b P 895c5b9da05444b28c3b3f9d31a20acd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:41:22.907015 3318 raft_consensus.cc:504] T 882624a8f1804980b7e6be4671b32c6b P 895c5b9da05444b28c3b3f9d31a20acd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895c5b9da05444b28c3b3f9d31a20acd" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 36919 } } peers { permanent_uuid: "c706c542cdf3492e95c148b302eb0ce8" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 34733 } } peers { permanent_uuid: "7489582e8a984290b3e9e68d847ef481" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 33251 } }
I20260925 10:41:22.908823 3318 leader_election.cc:288] T 882624a8f1804980b7e6be4671b32c6b P 895c5b9da05444b28c3b3f9d31a20acd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c706c542cdf3492e95c148b302eb0ce8 (127.23.100.2:34733), 7489582e8a984290b3e9e68d847ef481 (127.23.100.3:33251)
I20260925 10:41:22.915747 3309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:41:22.909352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:57113 (local address 127.23.100.3:33251)
0925 10:41:22.909559 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 10:41:22.909566 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:41:22.910276 (+ 710us) server_negotiation.cc:407] Connection header received
0925 10:41:22.910536 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:41:22.910545 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:41:22.910599 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:41:22.910688 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:41:22.911313 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:41:22.911960 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:41:22.914118 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:41:22.914885 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:41:22.915043 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:41:22.915109 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 10:41:22.915362 (+ 253us) server_negotiation.cc:299] Negotiation successful
0925 10:41:22.915480 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 10:41:22.915773 3310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:41:22.909095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:39907 (local address 127.23.100.2:34733)
0925 10:41:22.909309 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 10:41:22.909315 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:41:22.910245 (+ 930us) server_negotiation.cc:407] Connection header received
0925 10:41:22.910472 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:41:22.910477 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:41:22.910566 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:41:22.910663 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:41:22.911254 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:41:22.911919 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:41:22.914136 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:41:22.914900 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:41:22.915046 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:41:22.915117 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 10:41:22.915383 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 10:41:22.915496 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 10:41:22.915971 3321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:41:22.909304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.3:33251 (local address 127.23.100.1:57113)
0925 10:41:22.910168 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0925 10:41:22.910182 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:41:22.910360 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:41:22.910701 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:41:22.910705 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:41:22.910728 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:41:22.911148 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:41:22.911157 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:41:22.912487 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:41:22.912490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:41:22.913972 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:41:22.913986 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:41:22.915066 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:41:22.915072 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:41:22.915216 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:41:22.915237 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 10:41:22.915341 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 10:41:22.915497 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"mutex_wait_us":80,"thread_start_us":451,"threads_started":1}
I20260925 10:41:22.916009 3320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:41:22.909014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.2:34733 (local address 127.23.100.1:39907)
0925 10:41:22.910134 (+ 1120us) negotiation.cc:107] Waiting for socket to connect
0925 10:41:22.910148 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:41:22.910302 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:41:22.910679 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:41:22.910683 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:41:22.910714 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:41:22.911097 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:41:22.911105 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:41:22.912047 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:41:22.912055 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:41:22.913997 (+ 1942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:41:22.914008 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:41:22.915109 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:41:22.915114 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:41:22.915247 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:41:22.915266 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 10:41:22.915360 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 10:41:22.915487 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":923,"mutex_wait_us":72,"thread_start_us":725,"threads_started":1}
I20260925 10:41:22.916985 3113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "882624a8f1804980b7e6be4671b32c6b" candidate_uuid: "895c5b9da05444b28c3b3f9d31a20acd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c706c542cdf3492e95c148b302eb0ce8" is_pre_election: true
I20260925 10:41:22.917001 3247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "882624a8f1804980b7e6be4671b32c6b" candidate_uuid: "895c5b9da05444b28c3b3f9d31a20acd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7489582e8a984290b3e9e68d847ef481" is_pre_election: true
I20260925 10:41:22.917419 3113 raft_consensus.cc:2454] T 882624a8f1804980b7e6be4671b32c6b P c706c542cdf3492e95c148b302eb0ce8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 895c5b9da05444b28c3b3f9d31a20acd in term 0.
I20260925 10:41:22.917445 3247 raft_consensus.cc:2454] T 882624a8f1804980b7e6be4671b32c6b P 7489582e8a984290b3e9e68d847ef481 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 895c5b9da05444b28c3b3f9d31a20acd in term 0.
I20260925 10:41:22.918033 2905 leader_election.cc:302] T 882624a8f1804980b7e6be4671b32c6b P 895c5b9da05444b28c3b3f9d31a20acd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 895c5b9da05444b28c3b3f9d31a20acd, c706c542cdf3492e95c148b302eb0ce8; no voters:
I20260925 10:41:22.918395 3318 raft_consensus.cc:2781] T 882624a8f1804980b7e6be4671b32c6b P 895c5b9da05444b28c3b3f9d31a20acd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:41:22.918527 3318 raft_consensus.cc:482] T 882624a8f1804980b7e6be4671b32c6b P 895c5b9da05444b28c3b3f9d31a20acd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:41:22.918619 3318 raft_consensus.cc:3037] T 882624a8f1804980b7e6be4671b32c6b P 895c5b9da05444b28c3b3f9d31a20acd [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:41:22.920670 3318 raft_consensus.cc:504] T 882624a8f1804980b7e6be4671b32c6b P 895c5b9da05444b28c3b3f9d31a20acd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895c5b9da05444b28c3b3f9d31a20acd" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 36919 } } peers { permanent_uuid: "c706c542cdf3492e95c148b302eb0ce8" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 34733 } } peers { permanent_uuid: "7489582e8a984290b3e9e68d847ef481" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 33251 } }
I20260925 10:41:22.921288 3318 leader_election.cc:288] T 882624a8f1804980b7e6be4671b32c6b P 895c5b9da05444b28c3b3f9d31a20acd [CANDIDATE]: Term 1 election: Requested vote from peers c706c542cdf3492e95c148b302eb0ce8 (127.23.100.2:34733), 7489582e8a984290b3e9e68d847ef481 (127.23.100.3:33251)
I20260925 10:41:22.921626 3113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "882624a8f1804980b7e6be4671b32c6b" candidate_uuid: "895c5b9da05444b28c3b3f9d31a20acd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c706c542cdf3492e95c148b302eb0ce8"
I20260925 10:41:22.921756 3247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "882624a8f1804980b7e6be4671b32c6b" candidate_uuid: "895c5b9da05444b28c3b3f9d31a20acd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7489582e8a984290b3e9e68d847ef481"
I20260925 10:41:22.921823 3113 raft_consensus.cc:3037] T 882624a8f1804980b7e6be4671b32c6b P c706c542cdf3492e95c148b302eb0ce8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:41:22.921985 3247 raft_consensus.cc:3037] T 882624a8f1804980b7e6be4671b32c6b P 7489582e8a984290b3e9e68d847ef481 [term 0 FOLLOWER]: Advancing to term 1
W20260925 10:41:22.922612 3159 tablet.cc:2369] T 882624a8f1804980b7e6be4671b32c6b P c706c542cdf3492e95c148b302eb0ce8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:41:22.924594 3113 raft_consensus.cc:2454] T 882624a8f1804980b7e6be4671b32c6b P c706c542cdf3492e95c148b302eb0ce8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 895c5b9da05444b28c3b3f9d31a20acd in term 1.
I20260925 10:41:22.924973 2905 leader_election.cc:302] T 882624a8f1804980b7e6be4671b32c6b P 895c5b9da05444b28c3b3f9d31a20acd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 895c5b9da05444b28c3b3f9d31a20acd, c706c542cdf3492e95c148b302eb0ce8; no voters:
I20260925 10:41:22.925022 3247 raft_consensus.cc:2454] T 882624a8f1804980b7e6be4671b32c6b P 7489582e8a984290b3e9e68d847ef481 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 895c5b9da05444b28c3b3f9d31a20acd in term 1.
I20260925 10:41:22.925259 3318 raft_consensus.cc:2781] T 882624a8f1804980b7e6be4671b32c6b P 895c5b9da05444b28c3b3f9d31a20acd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:41:22.925652 3318 raft_consensus.cc:686] T 882624a8f1804980b7e6be4671b32c6b P 895c5b9da05444b28c3b3f9d31a20acd [term 1 LEADER]: Becoming Leader. State: Replica: 895c5b9da05444b28c3b3f9d31a20acd, State: Running, Role: LEADER
I20260925 10:41:22.926069 3318 consensus_queue.cc:237] T 882624a8f1804980b7e6be4671b32c6b P 895c5b9da05444b28c3b3f9d31a20acd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895c5b9da05444b28c3b3f9d31a20acd" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 36919 } } peers { permanent_uuid: "c706c542cdf3492e95c148b302eb0ce8" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 34733 } } peers { permanent_uuid: "7489582e8a984290b3e9e68d847ef481" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 33251 } }
I20260925 10:41:22.930563 2802 catalog_manager.cc:5495] T 882624a8f1804980b7e6be4671b32c6b P 895c5b9da05444b28c3b3f9d31a20acd reported cstate change: term changed from 0 to 1, leader changed from <none> to 895c5b9da05444b28c3b3f9d31a20acd (127.23.100.1). New cstate: current_term: 1 leader_uuid: "895c5b9da05444b28c3b3f9d31a20acd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895c5b9da05444b28c3b3f9d31a20acd" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 36919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c706c542cdf3492e95c148b302eb0ce8" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 34733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7489582e8a984290b3e9e68d847ef481" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 33251 } health_report { overall_health: UNKNOWN } } }
W20260925 10:41:22.946590 3016 tablet.cc:2369] T 882624a8f1804980b7e6be4671b32c6b P 895c5b9da05444b28c3b3f9d31a20acd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:41:22.948594 3306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:41:22.943585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33004 (local address 127.23.100.1:36919)
0925 10:41:22.943916 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0925 10:41:22.943923 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:41:22.943949 (+ 26us) server_negotiation.cc:407] Connection header received
0925 10:41:22.944018 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:41:22.944021 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:41:22.944093 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:41:22.944218 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:41:22.944523 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:41:22.945180 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:41:22.946549 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:41:22.947005 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:41:22.947093 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:41:22.947635 (+ 542us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:41:22.948076 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:41:22.948176 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 10:41:22.948271 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 10:41:22.948326 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163}
W20260925 10:41:22.951709 3293 tablet.cc:2369] T 882624a8f1804980b7e6be4671b32c6b P 7489582e8a984290b3e9e68d847ef481: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:41:22.956035 3247 raft_consensus.cc:1261] T 882624a8f1804980b7e6be4671b32c6b P 7489582e8a984290b3e9e68d847ef481 [term 1 FOLLOWER]: Refusing update from remote peer 895c5b9da05444b28c3b3f9d31a20acd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:41:22.956037 3113 raft_consensus.cc:1261] T 882624a8f1804980b7e6be4671b32c6b P c706c542cdf3492e95c148b302eb0ce8 [term 1 FOLLOWER]: Refusing update from remote peer 895c5b9da05444b28c3b3f9d31a20acd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:41:22.956933 3322 consensus_queue.cc:1035] T 882624a8f1804980b7e6be4671b32c6b P 895c5b9da05444b28c3b3f9d31a20acd [LEADER]: Connected to new peer: Peer: permanent_uuid: "7489582e8a984290b3e9e68d847ef481" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 33251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:41:22.957248 3318 consensus_queue.cc:1035] T 882624a8f1804980b7e6be4671b32c6b P 895c5b9da05444b28c3b3f9d31a20acd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c706c542cdf3492e95c148b302eb0ce8" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 34733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:41:22.973028 3326 mvcc.cc:205] Tried to move back new op lower bound from 7333203488578797568 to 7333203488468619264. Current Snapshot: MvccSnapshot[applied={T|T < 7333203488578797568 or (T in {7333203488578797568})}]
I20260925 10:41:22.981575 2801 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:41:22.981843 2801 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:41:22.985268 2801 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ca9a142d17bf4079a897f2db9419c51c: Sending DeleteTablet for 3 replicas of tablet 882624a8f1804980b7e6be4671b32c6b
I20260925 10:41:22.986276 2950 tablet_service.cc:1514] Processing DeleteTablet for tablet 882624a8f1804980b7e6be4671b32c6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:41:22 UTC) from {username='kudu', principal='kudu/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:32990
I20260925 10:41:22.986442 3093 tablet_service.cc:1514] Processing DeleteTablet for tablet 882624a8f1804980b7e6be4671b32c6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:41:22 UTC) from {username='kudu', principal='kudu/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:38714
I20260925 10:41:22.986552 3227 tablet_service.cc:1514] Processing DeleteTablet for tablet 882624a8f1804980b7e6be4671b32c6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:41:22 UTC) from {username='kudu', principal='kudu/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:49886
I20260925 10:41:22.989184 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 2882
I20260925 10:41:22.997243 3337 tablet_replica.cc:330] T 882624a8f1804980b7e6be4671b32c6b P 7489582e8a984290b3e9e68d847ef481: stopping tablet replica
I20260925 10:41:22.997713 3337 raft_consensus.cc:2229] T 882624a8f1804980b7e6be4671b32c6b P 7489582e8a984290b3e9e68d847ef481 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:41:22.998236 3337 raft_consensus.cc:2258] T 882624a8f1804980b7e6be4671b32c6b P 7489582e8a984290b3e9e68d847ef481 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:41:23.000594 3336 tablet_replica.cc:330] T 882624a8f1804980b7e6be4671b32c6b P c706c542cdf3492e95c148b302eb0ce8: stopping tablet replica
I20260925 10:41:23.001046 3336 raft_consensus.cc:2229] T 882624a8f1804980b7e6be4671b32c6b P c706c542cdf3492e95c148b302eb0ce8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:41:23.001590 3336 raft_consensus.cc:2258] T 882624a8f1804980b7e6be4671b32c6b P c706c542cdf3492e95c148b302eb0ce8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:41:23.004192 3337 ts_tablet_manager.cc:1905] T 882624a8f1804980b7e6be4671b32c6b P 7489582e8a984290b3e9e68d847ef481: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:41:23.004478 3336 ts_tablet_manager.cc:1905] T 882624a8f1804980b7e6be4671b32c6b P c706c542cdf3492e95c148b302eb0ce8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:41:23.007862 3335 tablet_replica.cc:330] T 882624a8f1804980b7e6be4671b32c6b P 895c5b9da05444b28c3b3f9d31a20acd: stopping tablet replica
I20260925 10:41:23.008229 3335 raft_consensus.cc:2229] T 882624a8f1804980b7e6be4671b32c6b P 895c5b9da05444b28c3b3f9d31a20acd [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:41:23.008889 3335 raft_consensus.cc:2258] T 882624a8f1804980b7e6be4671b32c6b P 895c5b9da05444b28c3b3f9d31a20acd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:41:23.011381 3337 ts_tablet_manager.cc:1918] T 882624a8f1804980b7e6be4671b32c6b P 7489582e8a984290b3e9e68d847ef481: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:41:23.011525 3337 log.cc:1196] T 882624a8f1804980b7e6be4671b32c6b P 7489582e8a984290b3e9e68d847ef481: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ts-2/wal/wals/882624a8f1804980b7e6be4671b32c6b
I20260925 10:41:23.012006 3337 ts_tablet_manager.cc:1939] T 882624a8f1804980b7e6be4671b32c6b P 7489582e8a984290b3e9e68d847ef481: Deleting consensus metadata
I20260925 10:41:23.013023 3336 ts_tablet_manager.cc:1918] T 882624a8f1804980b7e6be4671b32c6b P c706c542cdf3492e95c148b302eb0ce8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:41:23.013165 3336 log.cc:1196] T 882624a8f1804980b7e6be4671b32c6b P c706c542cdf3492e95c148b302eb0ce8: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ts-1/wal/wals/882624a8f1804980b7e6be4671b32c6b
I20260925 10:41:23.013631 3336 ts_tablet_manager.cc:1939] T 882624a8f1804980b7e6be4671b32c6b P c706c542cdf3492e95c148b302eb0ce8: Deleting consensus metadata
I20260925 10:41:23.013688 2787 catalog_manager.cc:4841] TS 7489582e8a984290b3e9e68d847ef481 (127.23.100.3:33251): tablet 882624a8f1804980b7e6be4671b32c6b (table test-table [id=820c731390ff434190e0676520d2e856]) successfully deleted
Sep 25 10:41:23 dist-test-slave-dxdd krb5kdc[2326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.1@KRBTEST.COM
I20260925 10:41:23.015148 2788 catalog_manager.cc:4841] TS c706c542cdf3492e95c148b302eb0ce8 (127.23.100.2:34733): tablet 882624a8f1804980b7e6be4671b32c6b (table test-table [id=820c731390ff434190e0676520d2e856]) successfully deleted
I20260925 10:41:23.015744 3335 ts_tablet_manager.cc:1905] T 882624a8f1804980b7e6be4671b32c6b P 895c5b9da05444b28c3b3f9d31a20acd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:41:23.022397 3306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:41:22.991250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33014 (local address 127.23.100.1:36919)
0925 10:41:22.991469 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 10:41:22.991477 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:41:23.002585 (+ 11108us) server_negotiation.cc:407] Connection header received
0925 10:41:23.002787 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:41:23.002796 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:41:23.002871 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:41:23.002960 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:41:23.004972 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:41:23.006579 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:41:23.008736 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:41:23.009409 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:41:23.009507 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:41:23.016173 (+ 6666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:41:23.016184 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:41:23.016195 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:41:23.016281 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:41:23.019655 (+ 3374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:41:23.020311 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:41:23.020319 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:41:23.020323 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:41:23.020493 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:41:23.020868 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:41:23.020874 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:41:23.020877 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:41:23.021209 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:41:23.021541 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0925 10:41:23.021784 (+ 243us) server_negotiation.cc:299] Negotiation successful
0925 10:41:23.022018 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 10:41:23.023195 3010 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:41:23.121589 3335 ts_tablet_manager.cc:1918] T 882624a8f1804980b7e6be4671b32c6b P 895c5b9da05444b28c3b3f9d31a20acd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:41:23.121861 3335 log.cc:1196] T 882624a8f1804980b7e6be4671b32c6b P 895c5b9da05444b28c3b3f9d31a20acd: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790332740062128-23952-0/minicluster-data/ts-0/wal/wals/882624a8f1804980b7e6be4671b32c6b
I20260925 10:41:23.122150 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 2882
W20260925 10:41:23.140017 2788 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.100.1:36919 (error 108)
I20260925 10:41:23.141175 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 3021
I20260925 10:41:23.141822 3308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:41:23.141363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:36919 (local address 127.0.0.1:33030)
0925 10:41:23.141526 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 10:41:23.141633 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.1:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 10:41:23.142127 2788 catalog_manager.cc:4584] TS 895c5b9da05444b28c3b3f9d31a20acd (127.23.100.1:36919): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 882624a8f1804980b7e6be4671b32c6b: Network error: Client connection negotiation failed: client connection to 127.23.100.1:36919: connect: Connection refused (error 111)
Sep 25 10:41:23 dist-test-slave-dxdd krb5kdc[2326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.2@KRBTEST.COM
I20260925 10:41:23.155375 3310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:41:23.141909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38722 (local address 127.23.100.2:34733)
0925 10:41:23.142078 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 10:41:23.142086 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:41:23.142109 (+ 23us) server_negotiation.cc:407] Connection header received
0925 10:41:23.142390 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:41:23.142395 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:41:23.142469 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:41:23.142545 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:41:23.143738 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:41:23.144434 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:41:23.145682 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:41:23.146045 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:41:23.146138 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:41:23.150041 (+ 3903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:41:23.150054 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:41:23.150067 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:41:23.150105 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:41:23.152747 (+ 2642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:41:23.153357 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:41:23.153363 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:41:23.153366 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:41:23.153484 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:41:23.153793 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:41:23.153796 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:41:23.153798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:41:23.154135 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:41:23.154478 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0925 10:41:23.154834 (+ 356us) server_negotiation.cc:299] Negotiation successful
0925 10:41:23.154998 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 10:41:23.156029 3153 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:41:23.195741 3308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:41:23.195287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:36919 (local address 127.0.0.1:33044)
0925 10:41:23.195488 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0925 10:41:23.195590 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.1:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 10:41:23.196729 3308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:41:23.196368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:36919 (local address 127.0.0.1:33054)
0925 10:41:23.196514 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 10:41:23.196601 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.1:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 10:41:23.243777 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 3021
I20260925 10:41:23.257288 3308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:41:23.256717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:36919 (local address 127.0.0.1:33070)
0925 10:41:23.256966 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0925 10:41:23.257079 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.1:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260925 10:41:23.258289 3308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:41:23.257982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:36919 (local address 127.0.0.1:33074)
0925 10:41:23.258138 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 10:41:23.258197 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.1:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 10:41:23.260679 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 3164
Sep 25 10:41:23 dist-test-slave-dxdd krb5kdc[2326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.3@KRBTEST.COM
I20260925 10:41:23.275023 3309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:41:23.261308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49898 (local address 127.23.100.3:33251)
0925 10:41:23.261469 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 10:41:23.261473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:41:23.261490 (+ 17us) server_negotiation.cc:407] Connection header received
0925 10:41:23.261582 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:41:23.261585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:41:23.261653 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:41:23.261728 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:41:23.263005 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:41:23.263709 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:41:23.264981 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:41:23.265334 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:41:23.265416 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:41:23.269806 (+ 4390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:41:23.269819 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:41:23.269835 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:41:23.269864 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:41:23.272648 (+ 2784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:41:23.273222 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:41:23.273229 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:41:23.273231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:41:23.273339 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:41:23.273627 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:41:23.273630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:41:23.273632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:41:23.273980 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:41:23.274171 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0925 10:41:23.274524 (+ 353us) server_negotiation.cc:299] Negotiation successful
0925 10:41:23.274669 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 10:41:23.275653 3287 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:41:23.338678 3308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:41:23.338217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:36919 (local address 127.0.0.1:33078)
0925 10:41:23.338440 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0925 10:41:23.338532 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.1:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260925 10:41:23.339711 3308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:41:23.339340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:36919 (local address 127.0.0.1:33080)
0925 10:41:23.339537 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 10:41:23.339611 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.1:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260925 10:41:23.363226 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 3164
I20260925 10:41:23.379293 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 2770
I20260925 10:41:23.380373 2832 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:41:23.490170 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 2770
I20260925 10:41:23.511180 23952 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 10:41:23.846664 23952 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 10:41:23.846875 23952 mini_ranger.cc:66] Stopping Ranger...
I20260925 10:41:24.191418 23952 mini_ranger.cc:68] Stopped Ranger
2026-09-25 10:41:24.191 UTC [2356] LOG: received smart shutdown request
2026-09-25 10:41:24.195 UTC [2356] LOG: background worker "logical replication launcher" (PID 2365) exited with exit code 1
2026-09-25 10:41:24.195 UTC [2360] LOG: shutting down
2026-09-25 10:41:24.337 UTC [2356] LOG: database system is shut down
2026-09-25T10:41:24Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47706 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:41:24 dist-test-slave-dxdd krb5kdc[3352](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:41:24 dist-test-slave-dxdd krb5kdc[3352](info): set up 2 sockets
Sep 25 10:41:24 dist-test-slave-dxdd krb5kdc[3352](info): commencing operation
krb5kdc: starting...
W20260925 10:41:25.570719 23952 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Sep 25 10:41:25 dist-test-slave-dxdd krb5kdc[3352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:41:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:41:25Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.23.100.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.23.100.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.23.100.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/rangeradmin_127.23.100.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.23.100.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/rangeradmin_127.23.100.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.23.100.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.23.100.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.23.100.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/rangerlookup_127.23.100.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.23.100.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/rangerlookup_127.23.100.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.100.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.100.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.100.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/HTTP_127.23.100.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.100.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/HTTP_127.23.100.20@KRBTEST.COM.keytab.
I20260925 10:41:25.878583 23952 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskVr7Mel/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/postgres -l logfile start
2026-09-25 10:41:28.179 UTC [3382] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 10:41:28.179 UTC [3382] LOG: listening on IPv4 address "127.23.100.20", port 45967
2026-09-25 10:41:28.182 UTC [3382] LOG: listening on Unix socket "/tmp/.s.PGSQL.45967"
2026-09-25 10:41:28.203 UTC [3385] LOG: database system was shut down at 2026-09-25 10:41:26 UTC
2026-09-25 10:41:28.207 UTC [3382] LOG: database system is ready to accept connections
I20260925 10:41:29.178570 23952 mini_postgres.cc:96] Postgres bound to 45967
2026-09-25 10:41:29.195 UTC [3393] FATAL: database "slave" does not exist
127.23.100.20:45967 - accepting connections
I20260925 10:41:29.196365 23952 mini_ranger.cc:161] Starting Ranger...
I20260925 10:41:29.221072 23952 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 10:41:29.659201 23952 mini_ranger.cc:87] Created ranger Postgres database
I20260925 10:41:29.659327 23952 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ranger-admin
2026-09-25 10:41:29,937 [I] DB FLAVOR :POSTGRES
2026-09-25 10:41:29,937 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 10:41:29,937 [I] Checking connection..
2026-09-25 10:41:29,937 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVr7Mel/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVr7Mel/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.20:45967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 10:41:30,168 [I] Checking connection passed.
2026-09-25 10:41:30,169 [I] --------- Verifying version history table ---------
2026-09-25 10:41:30,169 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVr7Mel/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVr7Mel/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.20:45967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 10:41:30,420 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 10:41:30,420 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVr7Mel/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVr7Mel/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.20:45967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 10:41:30,671 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 10:41:30,671 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 10:41:30,672 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVr7Mel/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVr7Mel/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.20:45967/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskVr7Mel/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 10:41:30.884 UTC [3457] WARNING: there is no transaction in progress
2026-09-25 10:41:30,890 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 10:41:30,891 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVr7Mel/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVr7Mel/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.20:45967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 10:41:31,138 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 10:41:31,138 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 10:41:31,138 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVr7Mel/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVr7Mel/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.20:45967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 10:41:31,366 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVr7Mel/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVr7Mel/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.20:45967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 10:41:31,597 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVr7Mel/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVr7Mel/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.20:45967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-dxdd.c.gcp-upstream.internal','N') ;"
2026-09-25 10:41:31,818 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVr7Mel/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVr7Mel/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.20:45967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 10:41:32,064 [I] Table x_portal_user does not exist in database ranger
2026-09-25 10:41:32,064 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVr7Mel/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVr7Mel/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.20:45967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 10:41:32,315 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 10:41:32,316 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVr7Mel/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVr7Mel/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.20:45967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 10:41:32,554 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVr7Mel/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVr7Mel/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.20:45967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 10:41:32,812 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 10:41:32,812 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVr7Mel/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVr7Mel/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.20:45967/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskVr7Mel/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 10:41:33.702 UTC [3565] WARNING: there is no transaction in progress
2026-09-25 10:41:33.720 UTC [3565] WARNING: there is no transaction in progress
2026-09-25 10:41:33.735 UTC [3565] WARNING: there is no transaction in progress
2026-09-25 10:41:33.750 UTC [3565] WARNING: there is no transaction in progress
2026-09-25 10:41:33.766 UTC [3565] WARNING: there is no transaction in progress
2026-09-25 10:41:33.782 UTC [3565] WARNING: there is no transaction in progress
2026-09-25 10:41:33.834 UTC [3565] WARNING: there is no transaction in progress
2026-09-25 10:41:33.840 UTC [3565] WARNING: there is no transaction in progress
2026-09-25 10:41:33.854 UTC [3565] WARNING: there is no transaction in progress
2026-09-25 10:41:33.864 UTC [3565] WARNING: there is no transaction in progress
2026-09-25 10:41:33.873 UTC [3565] WARNING: there is no transaction in progress
2026-09-25 10:41:33.885 UTC [3565] WARNING: there is no transaction in progress
2026-09-25 10:41:33.895 UTC [3565] WARNING: there is no transaction in progress
2026-09-25 10:41:33.901 UTC [3565] WARNING: there is no transaction in progress
2026-09-25 10:41:33.907 UTC [3565] WARNING: there is no transaction in progress
2026-09-25 10:41:33.914 UTC [3565] WARNING: there is no transaction in progress
2026-09-25 10:41:34,496 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 10:41:34,496 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVr7Mel/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVr7Mel/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.20:45967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 10:41:34,715 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 10:41:34,716 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVr7Mel/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVr7Mel/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.20:45967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 10:41:34,946 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 10:41:34,946 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVr7Mel/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVr7Mel/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.20:45967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 10:41:35,202 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 10:41:35,202 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVr7Mel/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVr7Mel/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.20:45967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 10:41:35,446 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVr7Mel/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVr7Mel/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.20:45967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 10:41:35,667 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVr7Mel/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVr7Mel/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.20:45967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-dxdd.c.gcp-upstream.internal';"
2026-09-25 10:41:35,906 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 10:41:35,907 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVr7Mel/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVr7Mel/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.20:45967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 10:41:36,154 [I] DB_PATCHES have already been applied
I20260925 10:41:36.160264 23952 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ranger-admin:/tmp/dist-test-taskVr7Mel/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskVr7Mel/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskVr7Mel/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVr7Mel/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 10:41:36.160344 23952 mini_ranger.cc:193] Using host: 127.23.100.20
I20260925 10:41:36.175232 23952 mini_ranger.cc:227] Ranger admin URL: http://127.23.100.20:45725
Sep 25, 2026 10:41:36 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 10:41:36 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 10:41:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 10:41:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 10:41:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 10:41:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 10:41:37 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 10:41:37.276026 23952 mini_ranger.cc:160] Time spent starting Ranger: real 8.080s user 0.000s sys 0.038s
Sep 25, 2026 10:41:55 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 10:41:55 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 25, 2026 10:41:55 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 10:41:55 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 10:41:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 10:41:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 10:41:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 10:41:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 10:41:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 10:41:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 10:41:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 10:41:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 10:41:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 10:41:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 10:41:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 10:41:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 10:41:55 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 10:41:56 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 10:41:57.265223 23952 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.23.100.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.23.100.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.23.100.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/rangerkms_127.23.100.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.23.100.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/rangerkms_127.23.100.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.100.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.23.100.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.100.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/HTTP_127.23.100.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.100.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/HTTP_127.23.100.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 10:41:57 dist-test-slave-dxdd krb5kdc[3352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332917, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 10:41:57.429032 23952 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 10:41:57.450100 23952 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 10:41:58.257537 23952 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 10:41:58.257654 23952 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ranger-kms
I20260925 10:41:58.257699 23952 mini_ranger_kms.cc:227] Using postgres at 127.23.100.20:45967
2026-09-25 10:41:58,600 [I] DB FLAVOR :POSTGRES
2026-09-25 10:41:58,600 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 10:41:58,600 [I] Checking connection
2026-09-25 10:41:58,600 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVr7Mel/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.20:45967/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 10:41:58,837 [I] connection success
2026-09-25 10:41:58,837 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 10:41:58,837 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 10:41:58,838 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVr7Mel/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.20:45967/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 10:41:59,092 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 10:41:59,092 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 10:41:59,092 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 10:41:59,092 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVr7Mel/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.20:45967/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 10:41:59,341 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 10:41:59.759146 23952 mini_ranger_kms.cc:310] Created kms service
I20260925 10:41:59.942478 23952 mini_ranger_kms.cc:326] Created kudu user
I20260925 10:41:59.984381 23952 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 10:41:59 dist-test-slave-dxdd krb5kdc[3352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332917, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.23.100.20@KRBTEST.COM
I20260925 10:42:00.357218 23952 mini_ranger_kms.cc:384] Added ranger policy
I20260925 10:42:00.357560 23952 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ranger-kms:/tmp/dist-test-taskVr7Mel/build/asan/bin/postgresql.jar:/tmp/dist-test-taskVr7Mel/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskVr7Mel/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 10:42:00.357618 23952 mini_ranger_kms.cc:240] Using host: 127.23.100.20
I20260925 10:42:00.374833 23952 mini_ranger_kms.cc:276] Ranger KMS PID: 3745
I20260925 10:42:00.374939 23952 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.23.100.20:34773
Sep 25, 2026 10:42:01 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 10:42:01 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 10:42:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 10:42:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 10:42:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 10:42:01 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 25 10:42:02 dist-test-slave-dxdd krb5kdc[3352](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790332922, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.100.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:42:02.595081 23952 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.166s user 0.110s sys 0.247s
I20260925 10:42:02.595341 23952 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 10:42:02.595507 23952 mini_ranger_kms.cc:401] 127.23.100.20:34773/kms/v1/keys
Sep 25 10:42:02 dist-test-slave-dxdd krb5kdc[3352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332922, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.100.20@KRBTEST.COM for HTTP/127.23.100.20@KRBTEST.COM
Sep 25 10:42:03 dist-test-slave-dxdd krb5kdc[3352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332922, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.100.20@KRBTEST.COM for HTTP/127.23.100.20@KRBTEST.COM
Sep 25 10:42:05 dist-test-slave-dxdd krb5kdc[3352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.23.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
I20260925 10:42:05.426002 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:46161
--webserver_interface=127.23.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:41949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.62:46161
--ranger_config_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.100.20:34773/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:42:05.756906 3796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:42:05.757181 3796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:42:05.757318 3796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:42:05.764755 3796 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:42:05.764851 3796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:42:05.764926 3796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:42:05.765034 3796 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:42:05.765100 3796 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:42:05.774886 3796 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:41949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/master-0/wal
--ranger_kms_url=127.23.100.20:34773/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.23.100.62:46161
--ranger_config_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:46161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:42:05.776105 3796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:42:05.777397 3796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:42:05.790844 3801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:42:05.790877 3802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:42:05.791718 3804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:42:05.792153 3796 server_base.cc:1034] running on GCE node
I20260925 10:42:05.793037 3796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:42:05.794919 3796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:42:05.796154 3796 hybrid_clock.cc:648] HybridClock initialized: now 1790332925796106 us; error 35 us; skew 500 ppm
I20260925 10:42:05.796520 3796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:42:05 dist-test-slave-dxdd krb5kdc[3352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332925, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:42:05.799626 3796 init.cc:377] Logged in from keytab as kudu/127.23.100.62@KRBTEST.COM (short username kudu)
I20260925 10:42:05.801123 3796 webserver.cc:460] Webserver started at http://127.23.100.62:34519/ using document root <none> and password file <none>
I20260925 10:42:05.801782 3796 fs_manager.cc:359] Metadata directory not provided
I20260925 10:42:05.801877 3796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:42:05.802210 3796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 10:42:05 dist-test-slave-dxdd krb5kdc[3352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332925, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.62@KRBTEST.COM for HTTP/127.23.100.20@KRBTEST.COM
I20260925 10:42:05.858235 3796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/master-0/data/instance:
uuid: "e8f9b2e6213b4dcba485041b6110ee68"
format_stamp: "Formatted at 2026-09-25 10:42:05 on dist-test-slave-dxdd"
server_key: "9678e13c468a6d9e6fd94f336ff8eb87"
server_key_iv: "cb725ce2b2a3f63c84cca8bf39a25dad"
server_key_version: "kuduclusterkey@0"
I20260925 10:42:05.859354 3796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/master-0/wal/instance:
uuid: "e8f9b2e6213b4dcba485041b6110ee68"
format_stamp: "Formatted at 2026-09-25 10:42:05 on dist-test-slave-dxdd"
server_key: "9678e13c468a6d9e6fd94f336ff8eb87"
server_key_iv: "cb725ce2b2a3f63c84cca8bf39a25dad"
server_key_version: "kuduclusterkey@0"
I20260925 10:42:05.866624 3796 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260925 10:42:05.886164 3811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:42:05.888060 3796 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 10:42:05.888271 3796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/master-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/master-0/wal
uuid: "e8f9b2e6213b4dcba485041b6110ee68"
format_stamp: "Formatted at 2026-09-25 10:42:05 on dist-test-slave-dxdd"
server_key: "9678e13c468a6d9e6fd94f336ff8eb87"
server_key_iv: "cb725ce2b2a3f63c84cca8bf39a25dad"
server_key_version: "kuduclusterkey@0"
I20260925 10:42:05.888533 3796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:42:05.913295 3796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:42:05.918181 3796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:42:05.918540 3796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:42:05.946916 3796 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.62:46161
I20260925 10:42:05.946902 3862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.62:46161 every 8 connection(s)
I20260925 10:42:05.948614 3796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
I20260925 10:42:05.949355 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 3796
I20260925 10:42:05.949579 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/master-0/wal/instance
I20260925 10:42:05.950438 23952 external_mini_cluster.cc:1428] Setting key bc52cb166ca047b445f3651945d2c1ad
I20260925 10:42:05.959735 3863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 10:42:05 dist-test-slave-dxdd krb5kdc[3352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:42:05.973119 3863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8f9b2e6213b4dcba485041b6110ee68: Bootstrap starting.
I20260925 10:42:05.976897 3863 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8f9b2e6213b4dcba485041b6110ee68: Neither blocks nor log segments found. Creating new log.
I20260925 10:42:05.977977 3863 log.cc:824] T 00000000000000000000000000000000 P e8f9b2e6213b4dcba485041b6110ee68: Log is configured to *not* fsync() on all Append() calls
I20260925 10:42:05.981528 3863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8f9b2e6213b4dcba485041b6110ee68: No bootstrap required, opened a new log
I20260925 10:42:05.983162 3866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:05.952767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49994 (local address 127.23.100.62:46161)
0925 10:42:05.958895 (+ 6128us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:05.958913 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:05.958964 (+ 51us) server_negotiation.cc:407] Connection header received
0925 10:42:05.961243 (+ 2279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:05.961272 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:05.961775 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:05.962221 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:42:05.963896 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:05.965878 (+ 1982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:05.967720 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:05.968771 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:05.968961 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:05.974022 (+ 5061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:42:05.974034 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:42:05.974054 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:42:05.974103 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:42:05.978075 (+ 3972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:05.978974 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:05.978980 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:05.978986 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:05.979147 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:05.979645 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:05.979650 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:05.979653 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:05.980298 (+ 645us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:42:05.980603 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:05.981319 (+ 716us) server_negotiation.cc:299] Negotiation successful
0925 10:42:05.981734 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5859,"thread_start_us":5606,"threads_started":1}
I20260925 10:42:05.990204 3863 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8f9b2e6213b4dcba485041b6110ee68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f9b2e6213b4dcba485041b6110ee68" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 46161 } }
I20260925 10:42:05.990703 3863 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8f9b2e6213b4dcba485041b6110ee68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:42:05.990830 3863 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8f9b2e6213b4dcba485041b6110ee68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8f9b2e6213b4dcba485041b6110ee68, State: Initialized, Role: FOLLOWER
I20260925 10:42:05.991448 3863 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8f9b2e6213b4dcba485041b6110ee68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f9b2e6213b4dcba485041b6110ee68" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 46161 } }
I20260925 10:42:05.991735 3863 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8f9b2e6213b4dcba485041b6110ee68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:42:05.991922 3863 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8f9b2e6213b4dcba485041b6110ee68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:42:05.992126 3863 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8f9b2e6213b4dcba485041b6110ee68 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:42:05.994626 3863 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8f9b2e6213b4dcba485041b6110ee68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f9b2e6213b4dcba485041b6110ee68" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 46161 } }
I20260925 10:42:05.995131 3863 leader_election.cc:302] T 00000000000000000000000000000000 P e8f9b2e6213b4dcba485041b6110ee68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8f9b2e6213b4dcba485041b6110ee68; no voters:
I20260925 10:42:05.995780 3863 leader_election.cc:288] T 00000000000000000000000000000000 P e8f9b2e6213b4dcba485041b6110ee68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:42:05.995893 3868 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8f9b2e6213b4dcba485041b6110ee68 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:42:05.998606 3868 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8f9b2e6213b4dcba485041b6110ee68 [term 1 LEADER]: Becoming Leader. State: Replica: e8f9b2e6213b4dcba485041b6110ee68, State: Running, Role: LEADER
I20260925 10:42:05.999377 3868 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8f9b2e6213b4dcba485041b6110ee68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f9b2e6213b4dcba485041b6110ee68" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 46161 } }
I20260925 10:42:06.000864 3863 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8f9b2e6213b4dcba485041b6110ee68 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:42:06.002295 3869 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8f9b2e6213b4dcba485041b6110ee68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8f9b2e6213b4dcba485041b6110ee68. Latest consensus state: current_term: 1 leader_uuid: "e8f9b2e6213b4dcba485041b6110ee68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f9b2e6213b4dcba485041b6110ee68" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 46161 } } }
I20260925 10:42:06.002786 3869 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8f9b2e6213b4dcba485041b6110ee68 [sys.catalog]: This master's current role is: LEADER
I20260925 10:42:06.002885 3870 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8f9b2e6213b4dcba485041b6110ee68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8f9b2e6213b4dcba485041b6110ee68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f9b2e6213b4dcba485041b6110ee68" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 46161 } } }
I20260925 10:42:06.003196 3870 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8f9b2e6213b4dcba485041b6110ee68 [sys.catalog]: This master's current role is: LEADER
I20260925 10:42:06.015386 3873 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:42:06.020780 3873 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:42:06.041131 3873 catalog_manager.cc:1347] Generated new cluster ID: a73bdab7ba7141998a319ee67ee86de8
I20260925 10:42:06.041258 3873 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:42:06.051332 3863 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 10:42:06.078334 3873 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:42:06.080344 3873 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:42:06.092788 3873 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8f9b2e6213b4dcba485041b6110ee68: Generated new TSK 0
I20260925 10:42:06.093963 3873 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab.
Entry for principal kudu/127.23.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab.
I20260925 10:42:06.917886 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.100.20:34773/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.1:0
--local_ip_for_outbound_sockets=127.23.100.1
--webserver_interface=127.23.100.1
--webserver_port=0
--tserver_master_addrs=127.23.100.62:46161
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab
--principal=kudu/127.23.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:41949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 10:42:07 dist-test-slave-dxdd krb5kdc[3352](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790332927, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 10:42:07.289778 3908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:42:07.290055 3908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:42:07.290210 3908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:42:07.297501 3908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:42:07.297638 3908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:42:07.297808 3908 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.1
I20260925 10:42:07.309398 3908 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:41949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.23.100.20:34773/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab
--principal=kudu/127.23.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.1
--webserver_port=0
--tserver_master_addrs=127.23.100.62:46161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.1
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:42:07.310811 3908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:42:07.312148 3908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:42:07.329092 3918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:42:07.330199 3919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:42:07.338213 3921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:42:07.338829 3908 server_base.cc:1034] running on GCE node
I20260925 10:42:07.339489 3908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:42:07.340547 3908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:42:07.341823 3908 hybrid_clock.cc:648] HybridClock initialized: now 1790332927341738 us; error 82 us; skew 500 ppm
I20260925 10:42:07.342144 3908 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:42:07 dist-test-slave-dxdd krb5kdc[3352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332927, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:42:07.345314 3908 init.cc:377] Logged in from keytab as kudu/127.23.100.1@KRBTEST.COM (short username kudu)
I20260925 10:42:07.351014 3908 webserver.cc:460] Webserver started at http://127.23.100.1:36133/ using document root <none> and password file <none>
I20260925 10:42:07.351759 3908 fs_manager.cc:359] Metadata directory not provided
I20260925 10:42:07.351895 3908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:42:07.352212 3908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 10:42:07 dist-test-slave-dxdd krb5kdc[3352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332927, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.1@KRBTEST.COM for HTTP/127.23.100.20@KRBTEST.COM
I20260925 10:42:07.371052 3908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-0/data/instance:
uuid: "1b102e5f357d4ef49c5c6703954d5b06"
format_stamp: "Formatted at 2026-09-25 10:42:07 on dist-test-slave-dxdd"
server_key: "eac72704d935bd4edeb51f469c59514e"
server_key_iv: "2e97289b78d8de97262949e64ad09b39"
server_key_version: "kuduclusterkey@0"
I20260925 10:42:07.372114 3908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-0/wal/instance:
uuid: "1b102e5f357d4ef49c5c6703954d5b06"
format_stamp: "Formatted at 2026-09-25 10:42:07 on dist-test-slave-dxdd"
server_key: "eac72704d935bd4edeb51f469c59514e"
server_key_iv: "2e97289b78d8de97262949e64ad09b39"
server_key_version: "kuduclusterkey@0"
I20260925 10:42:07.392314 3908 fs_manager.cc:691] Time spent creating directory manager: real 0.020s user 0.008s sys 0.012s
I20260925 10:42:07.427713 3928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:42:07.429816 3908 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.002s sys 0.007s
I20260925 10:42:07.430260 3908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-0/wal
uuid: "1b102e5f357d4ef49c5c6703954d5b06"
format_stamp: "Formatted at 2026-09-25 10:42:07 on dist-test-slave-dxdd"
server_key: "eac72704d935bd4edeb51f469c59514e"
server_key_iv: "2e97289b78d8de97262949e64ad09b39"
server_key_version: "kuduclusterkey@0"
I20260925 10:42:07.430685 3908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:42:07.475349 3908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:42:07.483731 3908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:42:07.484542 3908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:42:07.491971 3908 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:42:07.493985 3908 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:42:07.494093 3908 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:42:07.494232 3908 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:42:07.494329 3908 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 10:42:07 dist-test-slave-dxdd krb5kdc[3352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332927, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.62@KRBTEST.COM for HTTP/127.23.100.20@KRBTEST.COM
I20260925 10:42:07.829124 3908 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.1:37795
I20260925 10:42:07.829144 4040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.1:37795 every 8 connection(s)
I20260925 10:42:07.830488 3908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
I20260925 10:42:07.837749 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 3908
I20260925 10:42:07.837947 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-0/wal/instance
I20260925 10:42:07.838711 23952 external_mini_cluster.cc:1428] Setting key c0ed0d2ef31f9764f49f356cb6737b64
Sep 25 10:42:07 dist-test-slave-dxdd krb5kdc[3352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332927, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.1@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.100.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.2@KRBTEST.COM" created.
I20260925 10:42:07.886766 4043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:07.833744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:59953 (local address 127.23.100.62:46161)
0925 10:42:07.866692 (+ 32948us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:07.866698 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:07.866719 (+ 21us) server_negotiation.cc:407] Connection header received
0925 10:42:07.867828 (+ 1109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:07.867834 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:07.867927 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:07.868040 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:42:07.870342 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:07.871519 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:07.874022 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:07.874652 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:07.874750 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:07.879625 (+ 4875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:42:07.879633 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:42:07.879638 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:42:07.879678 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:42:07.882836 (+ 3158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:07.883736 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:07.883744 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:07.883750 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:07.883906 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:07.884437 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:07.884445 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:07.884464 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:07.884849 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:42:07.884974 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:07.885992 (+ 1018us) server_negotiation.cc:299] Negotiation successful
0925 10:42:07.886232 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32814,"thread_start_us":32704,"threads_started":1}
I20260925 10:42:07.889282 4044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:07.834451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:46161 (local address 127.23.100.1:59953)
0925 10:42:07.864428 (+ 29977us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:07.864497 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0925 10:42:07.867289 (+ 2792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:42:07.868489 (+ 1200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:42:07.868502 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:42:07.869226 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:42:07.870106 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:07.870131 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:07.871736 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:07.871743 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:07.873862 (+ 2119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:07.873877 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:07.875094 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:07.875103 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:07.875449 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:07.876336 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:42:07.876376 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:42:07.879309 (+ 2933us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:42:07.882993 (+ 3684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:42:07.883002 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:42:07.883019 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:42:07.883601 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:42:07.884046 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:42:07.884053 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:42:07.884060 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:42:07.884286 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:42:07.885075 (+ 789us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:42:07.885085 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:42:07.885514 (+ 429us) client_negotiation.cc:769] Sending connection context
0925 10:42:07.887331 (+ 1817us) client_negotiation.cc:240] Negotiation successful
0925 10:42:07.887893 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29618,"thread_start_us":39644,"threads_started":1}
I20260925 10:42:07.891372 4041 heartbeater.cc:347] Connected to a master server at 127.23.100.62:46161
I20260925 10:42:07.891697 4041 heartbeater.cc:464] Registering TS with master...
I20260925 10:42:07.892350 4041 heartbeater.cc:511] Master 127.23.100.62:46161 requested a full tablet report, sending...
I20260925 10:42:07.894634 3828 ts_manager.cc:194] Registered new tserver with Master: 1b102e5f357d4ef49c5c6703954d5b06 (127.23.100.1:37795)
I20260925 10:42:07.896703 3828 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.1@KRBTEST.COM'} at 127.23.100.1:59953
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab.
Entry for principal kudu/127.23.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab.
I20260925 10:42:07.917243 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.100.20:34773/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.2:0
--local_ip_for_outbound_sockets=127.23.100.2
--webserver_interface=127.23.100.2
--webserver_port=0
--tserver_master_addrs=127.23.100.62:46161
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab
--principal=kudu/127.23.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:41949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 10:42:08 dist-test-slave-dxdd krb5kdc[3352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332927, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.62@KRBTEST.COM for HTTP/127.23.100.20@KRBTEST.COM
W20260925 10:42:08.349699 4047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:42:08.350186 4047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:42:08.350435 4047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:42:08.362028 4047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:42:08.362233 4047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:42:08.362506 4047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.2
I20260925 10:42:08.377650 4047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:41949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.23.100.20:34773/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab
--principal=kudu/127.23.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.2
--webserver_port=0
--tserver_master_addrs=127.23.100.62:46161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.2
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:42:08.379222 4047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:42:08.380959 4047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:42:08.397073 4054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:42:08.397580 4055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:42:08.411327 4057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:42:08.411549 4047 server_base.cc:1034] running on GCE node
I20260925 10:42:08.414965 4047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:42:08.422617 4047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:42:08.426029 4047 hybrid_clock.cc:648] HybridClock initialized: now 1790332928425952 us; error 73 us; skew 500 ppm
I20260925 10:42:08.426498 4047 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:42:08 dist-test-slave-dxdd krb5kdc[3352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332928, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:42:08.430971 4047 init.cc:377] Logged in from keytab as kudu/127.23.100.2@KRBTEST.COM (short username kudu)
I20260925 10:42:08.432928 4047 webserver.cc:460] Webserver started at http://127.23.100.2:46009/ using document root <none> and password file <none>
I20260925 10:42:08.433900 4047 fs_manager.cc:359] Metadata directory not provided
I20260925 10:42:08.434064 4047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:42:08.434449 4047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 10:42:08 dist-test-slave-dxdd krb5kdc[3352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332928, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.2@KRBTEST.COM for HTTP/127.23.100.20@KRBTEST.COM
I20260925 10:42:08.450217 4047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-1/data/instance:
uuid: "ac3d0d1927d24613bddec74a922e7bf7"
format_stamp: "Formatted at 2026-09-25 10:42:08 on dist-test-slave-dxdd"
server_key: "01b37d1aec1df08fd69450210e9d90bb"
server_key_iv: "b750b0d1027a4bbdab15b2750c333619"
server_key_version: "kuduclusterkey@0"
I20260925 10:42:08.451315 4047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-1/wal/instance:
uuid: "ac3d0d1927d24613bddec74a922e7bf7"
format_stamp: "Formatted at 2026-09-25 10:42:08 on dist-test-slave-dxdd"
server_key: "01b37d1aec1df08fd69450210e9d90bb"
server_key_iv: "b750b0d1027a4bbdab15b2750c333619"
server_key_version: "kuduclusterkey@0"
I20260925 10:42:08.458119 4047 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260925 10:42:08.474464 4071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:42:08.476199 4047 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260925 10:42:08.476382 4047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-1/wal
uuid: "ac3d0d1927d24613bddec74a922e7bf7"
format_stamp: "Formatted at 2026-09-25 10:42:08 on dist-test-slave-dxdd"
server_key: "01b37d1aec1df08fd69450210e9d90bb"
server_key_iv: "b750b0d1027a4bbdab15b2750c333619"
server_key_version: "kuduclusterkey@0"
I20260925 10:42:08.476629 4047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:42:08.494395 4047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:42:08.499364 4047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:42:08.499699 4047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:42:08.500856 4047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:42:08.502812 4047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:42:08.502907 4047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:42:08.503079 4047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:42:08.503146 4047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:42:08.547153 4047 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.2:46423
I20260925 10:42:08.547178 4183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.2:46423 every 8 connection(s)
I20260925 10:42:08.548796 4047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
I20260925 10:42:08.552345 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 4047
I20260925 10:42:08.552552 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-1/wal/instance
I20260925 10:42:08.553260 23952 external_mini_cluster.cc:1428] Setting key 2b995730c637daa5fcbe7a0b24b7ba91
Sep 25 10:42:08 dist-test-slave-dxdd krb5kdc[3352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332928, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.2@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.100.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.3@KRBTEST.COM" created.
I20260925 10:42:08.587049 4187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:08.552243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:46161 (local address 127.23.100.2:47283)
0925 10:42:08.554624 (+ 2381us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:08.554682 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 10:42:08.557463 (+ 2781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:42:08.567286 (+ 9823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:42:08.567299 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:42:08.567918 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:42:08.568767 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:08.568794 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:08.570579 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:08.570587 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:08.572813 (+ 2226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:08.572831 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:08.573961 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:08.573969 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:08.574343 (+ 374us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:08.575106 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:42:08.575138 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:42:08.578067 (+ 2929us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:42:08.582173 (+ 4106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:42:08.582181 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:42:08.582198 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:42:08.582876 (+ 678us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:42:08.583312 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:42:08.583315 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:42:08.583317 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:42:08.583505 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:42:08.584222 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:42:08.584232 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:42:08.584635 (+ 403us) client_negotiation.cc:769] Sending connection context
0925 10:42:08.584923 (+ 288us) client_negotiation.cc:240] Negotiation successful
0925 10:42:08.585513 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2087,"thread_start_us":1840,"threads_started":1}
I20260925 10:42:08.591081 4186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:08.551712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:47283 (local address 127.23.100.62:46161)
0925 10:42:08.566688 (+ 14976us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:08.566696 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:08.566717 (+ 21us) server_negotiation.cc:407] Connection header received
0925 10:42:08.566795 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:08.566802 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:08.566904 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:08.568993 (+ 2089us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:42:08.569159 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:08.570400 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:08.573036 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:08.573723 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:08.574761 (+ 1038us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:08.578346 (+ 3585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:42:08.578354 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:42:08.578359 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:42:08.578404 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:42:08.582002 (+ 3598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:08.583007 (+ 1005us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:08.583012 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:08.583019 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:08.583128 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:08.583619 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:08.583624 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:08.583627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:08.583957 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:42:08.584110 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:08.590322 (+ 6212us) server_negotiation.cc:299] Negotiation successful
0925 10:42:08.590589 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14827,"thread_start_us":14700,"threads_started":1}
I20260925 10:42:08.592710 4184 heartbeater.cc:347] Connected to a master server at 127.23.100.62:46161
I20260925 10:42:08.593077 4184 heartbeater.cc:464] Registering TS with master...
I20260925 10:42:08.593936 4184 heartbeater.cc:511] Master 127.23.100.62:46161 requested a full tablet report, sending...
I20260925 10:42:08.595709 3828 ts_manager.cc:194] Registered new tserver with Master: ac3d0d1927d24613bddec74a922e7bf7 (127.23.100.2:46423)
I20260925 10:42:08.596936 3828 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.2@KRBTEST.COM'} at 127.23.100.2:47283
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab.
Entry for principal kudu/127.23.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab.
I20260925 10:42:08.615293 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.100.20:34773/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.3:0
--local_ip_for_outbound_sockets=127.23.100.3
--webserver_interface=127.23.100.3
--webserver_port=0
--tserver_master_addrs=127.23.100.62:46161
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab
--principal=kudu/127.23.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:41949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:42:08.901006 4041 heartbeater.cc:503] Master 127.23.100.62:46161 was elected leader, sending a full tablet report...
W20260925 10:42:08.944729 4190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:42:08.944999 4190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:42:08.945091 4190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:42:08.952374 4190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:42:08.952474 4190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:42:08.952646 4190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.3
I20260925 10:42:08.961998 4190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:41949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.23.100.20:34773/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab
--principal=kudu/127.23.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.3
--webserver_port=0
--tserver_master_addrs=127.23.100.62:46161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.3
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:42:08.963235 4190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:42:08.964447 4190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:42:08.977087 4195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:42:08.977135 4196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:42:08.977144 4198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:42:08.977701 4190 server_base.cc:1034] running on GCE node
I20260925 10:42:08.978261 4190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:42:08.979233 4190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:42:08.980417 4190 hybrid_clock.cc:648] HybridClock initialized: now 1790332928980370 us; error 37 us; skew 500 ppm
I20260925 10:42:08.980767 4190 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:42:08 dist-test-slave-dxdd krb5kdc[3352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332928, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:42:08.983892 4190 init.cc:377] Logged in from keytab as kudu/127.23.100.3@KRBTEST.COM (short username kudu)
I20260925 10:42:08.985206 4190 webserver.cc:460] Webserver started at http://127.23.100.3:38043/ using document root <none> and password file <none>
I20260925 10:42:08.985850 4190 fs_manager.cc:359] Metadata directory not provided
I20260925 10:42:08.985991 4190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:42:08.986280 4190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 10:42:08 dist-test-slave-dxdd krb5kdc[3352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332928, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.3@KRBTEST.COM for HTTP/127.23.100.20@KRBTEST.COM
I20260925 10:42:09.000278 4190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-2/data/instance:
uuid: "42798400926640c6a4caf15907941d2f"
format_stamp: "Formatted at 2026-09-25 10:42:08 on dist-test-slave-dxdd"
server_key: "0cc8d28a428c4d41284f3f6171215fe7"
server_key_iv: "30d013d8ab73589366986894ae1caf6d"
server_key_version: "kuduclusterkey@0"
I20260925 10:42:09.001116 4190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-2/wal/instance:
uuid: "42798400926640c6a4caf15907941d2f"
format_stamp: "Formatted at 2026-09-25 10:42:08 on dist-test-slave-dxdd"
server_key: "0cc8d28a428c4d41284f3f6171215fe7"
server_key_iv: "30d013d8ab73589366986894ae1caf6d"
server_key_version: "kuduclusterkey@0"
I20260925 10:42:09.006927 4190 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260925 10:42:09.023612 4205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:42:09.025243 4190 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 10:42:09.025413 4190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-2/wal
uuid: "42798400926640c6a4caf15907941d2f"
format_stamp: "Formatted at 2026-09-25 10:42:08 on dist-test-slave-dxdd"
server_key: "0cc8d28a428c4d41284f3f6171215fe7"
server_key_iv: "30d013d8ab73589366986894ae1caf6d"
server_key_version: "kuduclusterkey@0"
I20260925 10:42:09.025694 4190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:42:09.050261 4190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:42:09.061057 4190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:42:09.061357 4190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:42:09.062588 4190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:42:09.064203 4190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:42:09.064302 4190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:42:09.064414 4190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:42:09.064478 4190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:42:09.100490 4190 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.3:42537
I20260925 10:42:09.100443 4317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.3:42537 every 8 connection(s)
I20260925 10:42:09.102154 4190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
I20260925 10:42:09.107407 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 4190
I20260925 10:42:09.107569 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-2/wal/instance
I20260925 10:42:09.108094 23952 external_mini_cluster.cc:1428] Setting key 26e2f8a068a6676b0265154b5b0b75cd
Sep 25 10:42:09 dist-test-slave-dxdd krb5kdc[3352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332928, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.3@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:42:09.122403 4320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:09.104825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:55895 (local address 127.23.100.62:46161)
0925 10:42:09.105277 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:09.105281 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:09.106349 (+ 1068us) server_negotiation.cc:407] Connection header received
0925 10:42:09.108551 (+ 2202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:09.108557 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:09.108633 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:09.108734 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:42:09.110969 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:09.111932 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:09.113391 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:09.113798 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:09.113916 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:09.117007 (+ 3091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:42:09.117011 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:42:09.117014 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:42:09.117049 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:42:09.119613 (+ 2564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:09.120279 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:09.120284 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:09.120286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:09.120401 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:09.120806 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:09.120811 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:09.120814 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:09.121060 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:42:09.121208 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:09.121790 (+ 582us) server_negotiation.cc:299] Negotiation successful
0925 10:42:09.122007 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":238,"threads_started":1}
I20260925 10:42:09.123420 4321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:09.105496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:46161 (local address 127.23.100.3:55895)
0925 10:42:09.106176 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:09.106212 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 10:42:09.108234 (+ 2022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:42:09.109022 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:42:09.109034 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:42:09.109583 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:42:09.110243 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:09.110263 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:09.112067 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:09.112072 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:09.113253 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:09.113273 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:09.114006 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:09.114015 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:09.114242 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:09.114930 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:42:09.114950 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:42:09.116827 (+ 1877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:42:09.119749 (+ 2922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:42:09.119757 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:42:09.119770 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:42:09.120158 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:42:09.120503 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:42:09.120509 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:42:09.120512 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:42:09.120694 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:42:09.121183 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:42:09.121191 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:42:09.121468 (+ 277us) client_negotiation.cc:769] Sending connection context
0925 10:42:09.121704 (+ 236us) client_negotiation.cc:240] Negotiation successful
0925 10:42:09.122191 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":288,"threads_started":1}
I20260925 10:42:09.124864 4318 heartbeater.cc:347] Connected to a master server at 127.23.100.62:46161
I20260925 10:42:09.125177 4318 heartbeater.cc:464] Registering TS with master...
I20260925 10:42:09.125932 4318 heartbeater.cc:511] Master 127.23.100.62:46161 requested a full tablet report, sending...
I20260925 10:42:09.127336 3828 ts_manager.cc:194] Registered new tserver with Master: 42798400926640c6a4caf15907941d2f (127.23.100.3:42537)
I20260925 10:42:09.128108 3828 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.3@KRBTEST.COM'} at 127.23.100.3:55895
I20260925 10:42:09.136216 23952 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:42:09.138622 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 3908
Sep 25 10:42:09 dist-test-slave-dxdd krb5kdc[3352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.1@KRBTEST.COM
I20260925 10:42:09.159096 4323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:09.139515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50558 (local address 127.23.100.1:37795)
0925 10:42:09.142796 (+ 3281us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:09.142804 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:09.142828 (+ 24us) server_negotiation.cc:407] Connection header received
0925 10:42:09.142882 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:09.142888 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:09.143069 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:09.143195 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:42:09.145043 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:09.146028 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:09.148090 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:09.148586 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:09.148707 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:09.153558 (+ 4851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:42:09.153567 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:42:09.153579 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:42:09.153612 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:42:09.156350 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:09.157022 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:09.157028 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:09.157030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:09.157164 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:09.157513 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:09.157520 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:09.157523 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:09.157829 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:42:09.158120 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:09.158495 (+ 375us) server_negotiation.cc:299] Negotiation successful
0925 10:42:09.158774 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3130,"thread_start_us":2985,"threads_started":1}
I20260925 10:42:09.160398 4036 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:42:09.241981 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 3908
I20260925 10:42:09.258736 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 4047
Sep 25 10:42:09 dist-test-slave-dxdd krb5kdc[3352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.2@KRBTEST.COM
I20260925 10:42:09.273777 4325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:09.259601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60992 (local address 127.23.100.2:46423)
0925 10:42:09.260095 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:09.260102 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:09.260127 (+ 25us) server_negotiation.cc:407] Connection header received
0925 10:42:09.260183 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:09.260189 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:09.260392 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:09.260550 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:42:09.261723 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:09.262759 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:09.264052 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:09.264572 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:09.264705 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:09.268941 (+ 4236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:42:09.268950 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:42:09.268962 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:42:09.269015 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:42:09.271543 (+ 2528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:09.272141 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:09.272147 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:09.272149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:09.272270 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:09.272552 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:09.272555 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:09.272557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:09.272863 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:42:09.273065 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:09.273308 (+ 243us) server_negotiation.cc:299] Negotiation successful
0925 10:42:09.273485 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":260,"threads_started":1}
I20260925 10:42:09.275128 4179 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:42:09.354913 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 4047
I20260925 10:42:09.370776 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 4190
Sep 25 10:42:09 dist-test-slave-dxdd krb5kdc[3352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.3@KRBTEST.COM
I20260925 10:42:09.387015 4327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:09.371820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46176 (local address 127.23.100.3:42537)
0925 10:42:09.372559 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:09.372566 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:09.372595 (+ 29us) server_negotiation.cc:407] Connection header received
0925 10:42:09.372653 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:09.372659 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:09.372869 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:09.373004 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:42:09.374223 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:09.375256 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:09.376451 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:09.376971 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:09.377061 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:09.381543 (+ 4482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:42:09.381555 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:42:09.381570 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:42:09.381601 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:42:09.384240 (+ 2639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:09.384935 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:09.384943 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:09.384946 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:09.385079 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:09.385431 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:09.385437 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:09.385441 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:09.385803 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:42:09.386129 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:09.386407 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 10:42:09.386665 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":455,"threads_started":1}
I20260925 10:42:09.388262 4313 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:42:09.469659 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 4190
I20260925 10:42:09.485023 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 3796
I20260925 10:42:09.486220 3858 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:42:09.589527 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 3796
I20260925 10:42:09.610656 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:46161
--webserver_interface=127.23.100.62
--webserver_port=34519
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:41949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.62:46161
--ranger_config_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.100.20:34773/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.100.20:34773/kms with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:42:09.923002 4330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:42:09.923308 4330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:42:09.923480 4330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:42:09.930382 4330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:42:09.930538 4330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:42:09.930637 4330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:42:09.930735 4330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:42:09.930832 4330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:42:09.940635 4330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:41949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.100.20:34773/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.23.100.62:46161
--ranger_config_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:46161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.62
--webserver_port=34519
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:42:09.941876 4330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:42:09.943187 4330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:42:09.954651 4335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:42:09.954691 4338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:42:09.954591 4336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:42:09.955303 4330 server_base.cc:1034] running on GCE node
I20260925 10:42:09.956036 4330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:42:09.957334 4330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:42:09.958587 4330 hybrid_clock.cc:648] HybridClock initialized: now 1790332929958507 us; error 38 us; skew 500 ppm
I20260925 10:42:09.958910 4330 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:42:09 dist-test-slave-dxdd krb5kdc[3352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332929, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:42:09.962767 4330 init.cc:377] Logged in from keytab as kudu/127.23.100.62@KRBTEST.COM (short username kudu)
I20260925 10:42:09.964196 4330 webserver.cc:460] Webserver started at http://127.23.100.62:34519/ using document root <none> and password file <none>
I20260925 10:42:09.964815 4330 fs_manager.cc:359] Metadata directory not provided
I20260925 10:42:09.964955 4330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 10:42:10 dist-test-slave-dxdd krb5kdc[3352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332929, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.62@KRBTEST.COM for HTTP/127.23.100.20@KRBTEST.COM
I20260925 10:42:10.030292 4330 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 10:42:10.034026 4346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:42:10.035701 4330 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:42:10.035889 4330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/master-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/master-0/wal
uuid: "e8f9b2e6213b4dcba485041b6110ee68"
format_stamp: "Formatted at 2026-09-25 10:42:05 on dist-test-slave-dxdd"
server_key: "9678e13c468a6d9e6fd94f336ff8eb87"
server_key_iv: "cb725ce2b2a3f63c84cca8bf39a25dad"
server_key_version: "kuduclusterkey@0"
I20260925 10:42:10.037171 4330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:42:10.062479 4330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:42:10.067164 4330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:42:10.067461 4330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:42:10.086472 4330 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.62:46161
I20260925 10:42:10.086491 4397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.62:46161 every 8 connection(s)
I20260925 10:42:10.088227 4330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
I20260925 10:42:10.092787 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 4330
I20260925 10:42:10.093822 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.100.20:34773/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.1:37795
--local_ip_for_outbound_sockets=127.23.100.1
--tserver_master_addrs=127.23.100.62:46161
--webserver_port=36133
--webserver_interface=127.23.100.1
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab
--principal=kudu/127.23.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:41949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.100.20:34773/kms with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:42:10.095927 4398 sys_catalog.cc:263] Verifying existing consensus state
I20260925 10:42:10.101053 4398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8f9b2e6213b4dcba485041b6110ee68: Bootstrap starting.
I20260925 10:42:10.120607 4398 log.cc:824] T 00000000000000000000000000000000 P e8f9b2e6213b4dcba485041b6110ee68: Log is configured to *not* fsync() on all Append() calls
I20260925 10:42:10.129283 4398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8f9b2e6213b4dcba485041b6110ee68: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:42:10.129806 4398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8f9b2e6213b4dcba485041b6110ee68: Bootstrap complete.
I20260925 10:42:10.138572 4398 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8f9b2e6213b4dcba485041b6110ee68 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f9b2e6213b4dcba485041b6110ee68" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 46161 } }
I20260925 10:42:10.139552 4398 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8f9b2e6213b4dcba485041b6110ee68 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8f9b2e6213b4dcba485041b6110ee68, State: Initialized, Role: FOLLOWER
I20260925 10:42:10.140309 4398 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8f9b2e6213b4dcba485041b6110ee68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f9b2e6213b4dcba485041b6110ee68" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 46161 } }
I20260925 10:42:10.140539 4398 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8f9b2e6213b4dcba485041b6110ee68 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:42:10.140684 4398 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8f9b2e6213b4dcba485041b6110ee68 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:42:10.140866 4398 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8f9b2e6213b4dcba485041b6110ee68 [term 1 FOLLOWER]: Advancing to term 2
I20260925 10:42:10.145072 4398 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8f9b2e6213b4dcba485041b6110ee68 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f9b2e6213b4dcba485041b6110ee68" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 46161 } }
I20260925 10:42:10.145560 4398 leader_election.cc:302] T 00000000000000000000000000000000 P e8f9b2e6213b4dcba485041b6110ee68 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8f9b2e6213b4dcba485041b6110ee68; no voters:
I20260925 10:42:10.146256 4398 leader_election.cc:288] T 00000000000000000000000000000000 P e8f9b2e6213b4dcba485041b6110ee68 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 10:42:10.146406 4402 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8f9b2e6213b4dcba485041b6110ee68 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 10:42:10.154124 4402 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8f9b2e6213b4dcba485041b6110ee68 [term 2 LEADER]: Becoming Leader. State: Replica: e8f9b2e6213b4dcba485041b6110ee68, State: Running, Role: LEADER
I20260925 10:42:10.154749 4402 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8f9b2e6213b4dcba485041b6110ee68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f9b2e6213b4dcba485041b6110ee68" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 46161 } }
I20260925 10:42:10.154978 4398 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8f9b2e6213b4dcba485041b6110ee68 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:42:10.164062 4403 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8f9b2e6213b4dcba485041b6110ee68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e8f9b2e6213b4dcba485041b6110ee68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f9b2e6213b4dcba485041b6110ee68" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 46161 } } }
I20260925 10:42:10.164606 4403 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8f9b2e6213b4dcba485041b6110ee68 [sys.catalog]: This master's current role is: LEADER
I20260925 10:42:10.164951 4404 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8f9b2e6213b4dcba485041b6110ee68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8f9b2e6213b4dcba485041b6110ee68. Latest consensus state: current_term: 2 leader_uuid: "e8f9b2e6213b4dcba485041b6110ee68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f9b2e6213b4dcba485041b6110ee68" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 46161 } } }
I20260925 10:42:10.165345 4404 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8f9b2e6213b4dcba485041b6110ee68 [sys.catalog]: This master's current role is: LEADER
I20260925 10:42:10.176525 4408 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:42:10.181556 4408 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:42:10.184890 4408 catalog_manager.cc:1259] Loaded cluster ID: a73bdab7ba7141998a319ee67ee86de8
I20260925 10:42:10.186408 4408 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:42:10.190255 4408 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:42:10.195358 4408 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e8f9b2e6213b4dcba485041b6110ee68: Loaded TSK: 0
I20260925 10:42:10.197036 4408 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:42:10.291455 4398 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260925 10:42:10.529412 4400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:42:10.529819 4400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:42:10.530036 4400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:42:10.541751 4400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:42:10.541970 4400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:42:10.542176 4400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.1
I20260925 10:42:10.557626 4400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:41949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.100.20:34773/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab
--principal=kudu/127.23.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.1:37795
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.1
--webserver_port=36133
--tserver_master_addrs=127.23.100.62:46161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.1
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:42:10.559432 4400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:42:10.561179 4400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:42:10.611692 4435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:42:10.612315 4436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:42:10.626718 4400 server_base.cc:1034] running on GCE node
W20260925 10:42:10.630606 4439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:42:10.631543 4400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:42:10.632887 4400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:42:10.634182 4400 hybrid_clock.cc:648] HybridClock initialized: now 1790332930634106 us; error 62 us; skew 500 ppm
I20260925 10:42:10.634581 4400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:42:10 dist-test-slave-dxdd krb5kdc[3352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332930, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:42:10.639093 4400 init.cc:377] Logged in from keytab as kudu/127.23.100.1@KRBTEST.COM (short username kudu)
I20260925 10:42:10.649317 4400 webserver.cc:460] Webserver started at http://127.23.100.1:36133/ using document root <none> and password file <none>
I20260925 10:42:10.650427 4400 fs_manager.cc:359] Metadata directory not provided
I20260925 10:42:10.650674 4400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 10:42:10 dist-test-slave-dxdd krb5kdc[3352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332930, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.1@KRBTEST.COM for HTTP/127.23.100.20@KRBTEST.COM
I20260925 10:42:10.693666 4400 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.005s sys 0.001s
I20260925 10:42:10.702858 4446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:42:10.704859 4400 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.006s sys 0.000s
I20260925 10:42:10.705113 4400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-0/wal
uuid: "1b102e5f357d4ef49c5c6703954d5b06"
format_stamp: "Formatted at 2026-09-25 10:42:07 on dist-test-slave-dxdd"
server_key: "eac72704d935bd4edeb51f469c59514e"
server_key_iv: "2e97289b78d8de97262949e64ad09b39"
server_key_version: "kuduclusterkey@0"
I20260925 10:42:10.706780 4400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:42:10.730127 4400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:42:10.735384 4400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:42:10.735776 4400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:42:10.746444 4400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:42:10.748358 4400 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:42:10.748530 4400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260925 10:42:10.748704 4400 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:42:10.748819 4400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:42:10.830693 4400 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.1:37795
I20260925 10:42:10.830746 4558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.1:37795 every 8 connection(s)
I20260925 10:42:10.832378 4400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
I20260925 10:42:10.843391 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 4400
I20260925 10:42:10.844202 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.100.20:34773/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.2:46423
--local_ip_for_outbound_sockets=127.23.100.2
--tserver_master_addrs=127.23.100.62:46161
--webserver_port=46009
--webserver_interface=127.23.100.2
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab
--principal=kudu/127.23.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:41949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.100.20:34773/kms with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 10:42:10 dist-test-slave-dxdd krb5kdc[3352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332930, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.1@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:42:10.879623 4562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:10.839076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:33755 (local address 127.23.100.62:46161)
0925 10:42:10.844535 (+ 5459us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:10.844555 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:10.844606 (+ 51us) server_negotiation.cc:407] Connection header received
0925 10:42:10.845057 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:10.845077 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:10.845587 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:10.848063 (+ 2476us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:42:10.848532 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:10.850729 (+ 2197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:10.853215 (+ 2486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:10.854269 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:10.854513 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:10.858958 (+ 4445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:42:10.858970 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:42:10.858991 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:42:10.859039 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:42:10.863261 (+ 4222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:10.874938 (+ 11677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:10.874950 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:10.874959 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:10.875209 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:10.875881 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:10.875886 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:10.875890 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:10.876572 (+ 682us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:42:10.876745 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:10.877805 (+ 1060us) server_negotiation.cc:299] Negotiation successful
0925 10:42:10.878240 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5181,"thread_start_us":4967,"threads_started":1}
I20260925 10:42:10.882089 4561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:10.837763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:46161 (local address 127.23.100.1:33755)
0925 10:42:10.838958 (+ 1195us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:10.839025 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0925 10:42:10.843655 (+ 4630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:42:10.846239 (+ 2584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:42:10.846251 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:42:10.846902 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:42:10.847741 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:10.847771 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:10.850968 (+ 3197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:10.850974 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:10.853017 (+ 2043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:10.853032 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:10.854593 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:10.854601 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:10.854930 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:10.855745 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:42:10.855776 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:42:10.858652 (+ 2876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:42:10.865086 (+ 6434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:42:10.865098 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:42:10.865117 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:42:10.874720 (+ 9603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:42:10.875403 (+ 683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:42:10.875410 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:42:10.875416 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:42:10.875724 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:42:10.876846 (+ 1122us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:42:10.876854 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:42:10.877252 (+ 398us) client_negotiation.cc:769] Sending connection context
0925 10:42:10.880481 (+ 3229us) client_negotiation.cc:240] Negotiation successful
0925 10:42:10.880986 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":767,"thread_start_us":1627,"threads_started":1}
I20260925 10:42:10.885401 4559 heartbeater.cc:347] Connected to a master server at 127.23.100.62:46161
I20260925 10:42:10.885733 4559 heartbeater.cc:464] Registering TS with master...
I20260925 10:42:10.886624 4559 heartbeater.cc:511] Master 127.23.100.62:46161 requested a full tablet report, sending...
W20260925 10:42:10.889858 4559 heartbeater.cc:644] Failed to heartbeat to 127.23.100.62:46161 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 10:42:10.890064 4559 heartbeater.cc:464] Registering TS with master...
I20260925 10:42:10.890336 4559 heartbeater.cc:511] Master 127.23.100.62:46161 requested a full tablet report, sending...
I20260925 10:42:10.891541 4559 heartbeater.cc:464] Registering TS with master...
I20260925 10:42:10.891783 4559 heartbeater.cc:511] Master 127.23.100.62:46161 requested a full tablet report, sending...
W20260925 10:42:10.892804 4559 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 10:42:11.232338 4563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:42:11.232635 4563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:42:11.232806 4563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:42:11.240305 4563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:42:11.240494 4563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:42:11.240700 4563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.2
I20260925 10:42:11.251124 4563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:41949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.100.20:34773/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab
--principal=kudu/127.23.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.2:46423
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.2
--webserver_port=46009
--tserver_master_addrs=127.23.100.62:46161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.2
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:42:11.252437 4563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:42:11.253765 4563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:42:11.271003 4576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:42:11.271039 4577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:42:11.279107 4580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:42:11.279908 4563 server_base.cc:1034] running on GCE node
I20260925 10:42:11.280687 4563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:42:11.281981 4563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:42:11.283257 4563 hybrid_clock.cc:648] HybridClock initialized: now 1790332931283180 us; error 61 us; skew 500 ppm
I20260925 10:42:11.283623 4563 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:42:11 dist-test-slave-dxdd krb5kdc[3352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332931, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:42:11.287815 4563 init.cc:377] Logged in from keytab as kudu/127.23.100.2@KRBTEST.COM (short username kudu)
I20260925 10:42:11.290498 4563 webserver.cc:460] Webserver started at http://127.23.100.2:46009/ using document root <none> and password file <none>
I20260925 10:42:11.291426 4563 fs_manager.cc:359] Metadata directory not provided
I20260925 10:42:11.291600 4563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 10:42:11 dist-test-slave-dxdd krb5kdc[3352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332931, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.2@KRBTEST.COM for HTTP/127.23.100.20@KRBTEST.COM
I20260925 10:42:11.312656 4563 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.003s
I20260925 10:42:11.315886 4587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:42:11.317317 4563 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:42:11.317488 4563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-1/wal
uuid: "ac3d0d1927d24613bddec74a922e7bf7"
format_stamp: "Formatted at 2026-09-25 10:42:08 on dist-test-slave-dxdd"
server_key: "01b37d1aec1df08fd69450210e9d90bb"
server_key_iv: "b750b0d1027a4bbdab15b2750c333619"
server_key_version: "kuduclusterkey@0"
I20260925 10:42:11.318542 4563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:42:11.329725 4563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:42:11.333703 4563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:42:11.333976 4563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:42:11.335075 4563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:42:11.336479 4563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:42:11.336599 4563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:42:11.336714 4563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:42:11.336793 4563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:42:11.379971 4563 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.2:46423
I20260925 10:42:11.380023 4699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.2:46423 every 8 connection(s)
I20260925 10:42:11.381435 4563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
I20260925 10:42:11.386053 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 4563
I20260925 10:42:11.386890 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.100.20:34773/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.3:42537
--local_ip_for_outbound_sockets=127.23.100.3
--tserver_master_addrs=127.23.100.62:46161
--webserver_port=38043
--webserver_interface=127.23.100.3
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab
--principal=kudu/127.23.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:41949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.100.20:34773/kms with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 10:42:11 dist-test-slave-dxdd krb5kdc[3352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332931, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.2@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:42:11.410264 4702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:11.384432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:54751 (local address 127.23.100.62:46161)
0925 10:42:11.389697 (+ 5265us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:11.389705 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:11.389727 (+ 22us) server_negotiation.cc:407] Connection header received
0925 10:42:11.389790 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:11.389797 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:11.389895 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:11.390017 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:42:11.392022 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:11.393086 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:11.395615 (+ 2529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:11.396344 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:11.396482 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:11.402541 (+ 6059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:42:11.402548 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:42:11.402554 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:42:11.402604 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:42:11.406217 (+ 3613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:11.407203 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:11.407210 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:11.407213 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:11.407364 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:11.407882 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:11.407890 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:11.407896 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:11.408298 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:42:11.408436 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:11.409519 (+ 1083us) server_negotiation.cc:299] Negotiation successful
0925 10:42:11.409783 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1110,"thread_start_us":969,"threads_started":1}
I20260925 10:42:11.413136 4703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:11.385150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:46161 (local address 127.23.100.2:54751)
0925 10:42:11.386367 (+ 1217us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:11.386423 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 10:42:11.389229 (+ 2806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:42:11.390236 (+ 1007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:42:11.390244 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:42:11.390889 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:42:11.391759 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:11.391794 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:11.393244 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:11.393251 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:11.395395 (+ 2144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:11.395419 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:11.396768 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:11.396773 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:11.396970 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:11.397699 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:42:11.397722 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:42:11.402254 (+ 4532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:42:11.406402 (+ 4148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:42:11.406413 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:42:11.406433 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:42:11.407049 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:42:11.407500 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:42:11.407507 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:42:11.407513 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:42:11.407745 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:42:11.408545 (+ 800us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:42:11.408573 (+ 28us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:42:11.409019 (+ 446us) client_negotiation.cc:769] Sending connection context
0925 10:42:11.410893 (+ 1874us) client_negotiation.cc:240] Negotiation successful
0925 10:42:11.411508 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":889,"thread_start_us":628,"threads_started":1}
I20260925 10:42:11.415370 4700 heartbeater.cc:347] Connected to a master server at 127.23.100.62:46161
I20260925 10:42:11.415740 4700 heartbeater.cc:464] Registering TS with master...
I20260925 10:42:11.416646 4700 heartbeater.cc:511] Master 127.23.100.62:46161 requested a full tablet report, sending...
I20260925 10:42:11.418720 4363 ts_manager.cc:194] Registered new tserver with Master: ac3d0d1927d24613bddec74a922e7bf7 (127.23.100.2:46423)
I20260925 10:42:11.420855 4363 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.2@KRBTEST.COM'} at 127.23.100.2:54751
Sep 25 10:42:11 dist-test-slave-dxdd krb5kdc[3352](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790332931, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 10:42:11.813711 4706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:42:11.813967 4706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:42:11.814118 4706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:42:11.823122 4706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:42:11.823223 4706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:42:11.823414 4706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.3
I20260925 10:42:11.838572 4706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:41949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.100.20:34773/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab
--principal=kudu/127.23.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.3:42537
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.3
--webserver_port=38043
--tserver_master_addrs=127.23.100.62:46161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.3
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:42:11.840396 4706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:42:11.842329 4706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:42:11.859620 4714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:42:11.859939 4715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:42:11.867430 4706 server_base.cc:1034] running on GCE node
W20260925 10:42:11.868106 4717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:42:11.868949 4706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:42:11.873870 4706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:42:11.875167 4706 hybrid_clock.cc:648] HybridClock initialized: now 1790332931875089 us; error 62 us; skew 500 ppm
I20260925 10:42:11.875578 4706 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:42:11 dist-test-slave-dxdd krb5kdc[3352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332931, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:42:11.879976 4706 init.cc:377] Logged in from keytab as kudu/127.23.100.3@KRBTEST.COM (short username kudu)
I20260925 10:42:11.885146 4706 webserver.cc:460] Webserver started at http://127.23.100.3:38043/ using document root <none> and password file <none>
I20260925 10:42:11.886094 4706 fs_manager.cc:359] Metadata directory not provided
I20260925 10:42:11.886252 4706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:42:11.892956 4559 heartbeater.cc:464] Registering TS with master...
I20260925 10:42:11.893371 4559 heartbeater.cc:511] Master 127.23.100.62:46161 requested a full tablet report, sending...
I20260925 10:42:11.894609 4363 ts_manager.cc:194] Registered new tserver with Master: 1b102e5f357d4ef49c5c6703954d5b06 (127.23.100.1:37795)
Sep 25 10:42:11 dist-test-slave-dxdd krb5kdc[3352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332931, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.3@KRBTEST.COM for HTTP/127.23.100.20@KRBTEST.COM
I20260925 10:42:11.895752 4363 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.1@KRBTEST.COM'} at 127.23.100.1:33755
I20260925 10:42:11.917059 4706 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.001s sys 0.004s
I20260925 10:42:11.920374 4725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:42:11.921947 4706 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:42:11.922118 4706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-2/wal
uuid: "42798400926640c6a4caf15907941d2f"
format_stamp: "Formatted at 2026-09-25 10:42:08 on dist-test-slave-dxdd"
server_key: "0cc8d28a428c4d41284f3f6171215fe7"
server_key_iv: "30d013d8ab73589366986894ae1caf6d"
server_key_version: "kuduclusterkey@0"
I20260925 10:42:11.923254 4706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:42:11.975164 4706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:42:11.979565 4706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:42:11.979859 4706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:42:11.981128 4706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:42:11.995136 4706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:42:11.995282 4706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:42:11.995396 4706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:42:11.995472 4706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 10:42:12 dist-test-slave-dxdd krb5kdc[3352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332931, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.62@KRBTEST.COM for HTTP/127.23.100.20@KRBTEST.COM
I20260925 10:42:12.118688 4706 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.3:42537
I20260925 10:42:12.118770 4837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.3:42537 every 8 connection(s)
I20260925 10:42:12.120514 4706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
I20260925 10:42:12.126108 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 4706
Sep 25 10:42:12 dist-test-slave-dxdd krb5kdc[3352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332931, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.3@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:42:12.187373 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:12.127052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39696 (local address 127.23.100.62:46161)
0925 10:42:12.159136 (+ 32084us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:12.159143 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:12.162983 (+ 3840us) server_negotiation.cc:407] Connection header received
0925 10:42:12.164784 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:12.164791 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:12.164895 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:12.164975 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:42:12.170679 (+ 5704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:12.171950 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:12.173999 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:12.174765 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:12.174873 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:12.179237 (+ 4364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:42:12.179243 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:42:12.179247 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:42:12.179284 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:42:12.184010 (+ 4726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:12.184873 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:12.184879 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:12.184883 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:12.185035 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:12.185504 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:12.185509 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:12.185512 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:12.185856 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:42:12.186125 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:12.186762 (+ 637us) server_negotiation.cc:299] Negotiation successful
0925 10:42:12.186985 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":31955,"thread_start_us":30110,"threads_started":1}
I20260925 10:42:12.188542 4842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:12.124479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:60859 (local address 127.23.100.62:46161)
0925 10:42:12.157135 (+ 32656us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:12.157144 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:12.162350 (+ 5206us) server_negotiation.cc:407] Connection header received
0925 10:42:12.166428 (+ 4078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:12.166434 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:12.166529 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:12.166604 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:42:12.170459 (+ 3855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:12.171697 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:12.174145 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:12.174951 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:12.175068 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:12.179695 (+ 4627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:42:12.179702 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:42:12.179707 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:42:12.179760 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:42:12.183782 (+ 4022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:12.184661 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:12.184667 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:12.184670 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:12.184802 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:12.185311 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:12.185316 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:12.185320 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:12.185661 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:42:12.185783 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:12.186789 (+ 1006us) server_negotiation.cc:299] Negotiation successful
0925 10:42:12.188180 (+ 1391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1246,"server-negotiator.queue_time_us":32482,"thread_start_us":34641,"threads_started":1}
I20260925 10:42:12.188748 4840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:12.123902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:46161 (local address 127.23.100.3:60859)
0925 10:42:12.159867 (+ 35965us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:12.159938 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0925 10:42:12.164443 (+ 4505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:42:12.167213 (+ 2770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:42:12.167226 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:42:12.167935 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:42:12.168813 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:12.168839 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:12.171910 (+ 3071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:12.171916 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:12.173951 (+ 2035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:12.173974 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:12.175368 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:12.175374 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:12.175665 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:12.176400 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:42:12.176429 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:42:12.179358 (+ 2929us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:42:12.183942 (+ 4584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:42:12.183951 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:42:12.183970 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:42:12.184527 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:42:12.184934 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:42:12.184938 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:42:12.184942 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:42:12.185184 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:42:12.185877 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:42:12.185885 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:42:12.186315 (+ 430us) client_negotiation.cc:769] Sending connection context
0925 10:42:12.186901 (+ 586us) client_negotiation.cc:240] Negotiation successful
0925 10:42:12.187431 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35615,"thread_start_us":39002,"threads_started":1}
I20260925 10:42:12.190699 4838 heartbeater.cc:347] Connected to a master server at 127.23.100.62:46161
I20260925 10:42:12.191035 4838 heartbeater.cc:464] Registering TS with master...
I20260925 10:42:12.191920 4838 heartbeater.cc:511] Master 127.23.100.62:46161 requested a full tablet report, sending...
I20260925 10:42:12.193655 4363 ts_manager.cc:194] Registered new tserver with Master: 42798400926640c6a4caf15907941d2f (127.23.100.3:42537)
I20260925 10:42:12.194808 4363 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.3@KRBTEST.COM'} at 127.23.100.3:60859
I20260925 10:42:12.196477 23952 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:42:12.219766 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:12.203741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39702 (local address 127.23.100.62:46161)
0925 10:42:12.203896 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:12.203902 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:12.204395 (+ 493us) server_negotiation.cc:407] Connection header received
0925 10:42:12.204563 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:12.204569 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:12.204649 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:12.204740 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:42:12.206698 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:12.207754 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:12.209665 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:12.210279 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:12.210406 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:12.213625 (+ 3219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:42:12.213632 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:42:12.213637 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:42:12.213672 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:42:12.217002 (+ 3330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:12.217787 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:12.217792 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:12.217796 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:12.217929 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:12.218350 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:12.218354 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:12.218358 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:12.218683 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:42:12.218905 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:12.219213 (+ 308us) server_negotiation.cc:299] Negotiation successful
0925 10:42:12.219429 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 10:42:12.225260 4363 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39702:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:42:12.236217 4363 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:42:12.288435 4853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:12.255100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:37795 (local address 127.0.0.1:53598)
0925 10:42:12.255823 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:12.255861 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 10:42:12.256026 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:42:12.280149 (+ 24123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:42:12.280154 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:42:12.280178 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:42:12.280477 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:12.280513 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:12.282638 (+ 2125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:12.282644 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:12.285197 (+ 2553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:12.285213 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:12.287572 (+ 2359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:12.287579 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:12.287778 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:12.287809 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 10:42:12.287879 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 10:42:12.288019 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"mutex_wait_us":73,"thread_start_us":314,"threads_started":1}
I20260925 10:42:12.288712 4858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:12.259226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47784 (local address 127.23.100.3:42537)
0925 10:42:12.277886 (+ 18660us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:12.277899 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:12.278757 (+ 858us) server_negotiation.cc:407] Connection header received
0925 10:42:12.278926 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:12.278933 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:12.279151 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:12.279328 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:42:12.280185 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:12.281661 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:12.285355 (+ 3694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:12.286741 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:12.286870 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:12.287098 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:12.288134 (+ 1036us) server_negotiation.cc:299] Negotiation successful
0925 10:42:12.288328 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18444,"thread_start_us":18685,"threads_started":1}
I20260925 10:42:12.290961 4854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:12.255392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53598 (local address 127.23.100.1:37795)
0925 10:42:12.279430 (+ 24038us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:12.279439 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:12.279470 (+ 31us) server_negotiation.cc:407] Connection header received
0925 10:42:12.279575 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:12.279585 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:12.279803 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:12.280681 (+ 878us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:42:12.280909 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:12.282477 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:12.285385 (+ 2908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:12.286813 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:12.290154 (+ 3341us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:12.290402 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:12.290470 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 10:42:12.290636 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23866,"thread_start_us":23700,"threads_started":1}
I20260925 10:42:12.291524 4855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:12.258544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.3:42537 (local address 127.0.0.1:47784)
0925 10:42:12.278627 (+ 20083us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:12.278645 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:42:12.278772 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:42:12.279626 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:42:12.279630 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:42:12.279653 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:42:12.279962 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:12.279972 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:12.281824 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:12.281829 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:12.285180 (+ 3351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:12.285194 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:12.287254 (+ 2060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:12.287260 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:12.287534 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:12.287646 (+ 112us) client_negotiation.cc:769] Sending connection context
0925 10:42:12.287792 (+ 146us) client_negotiation.cc:240] Negotiation successful
0925 10:42:12.288015 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19888,"mutex_wait_us":125,"thread_start_us":17455,"threads_started":1}
I20260925 10:42:12.293924 4857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:12.257001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.2:46423 (local address 127.0.0.1:49410)
0925 10:42:12.276041 (+ 19040us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:12.276076 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 10:42:12.276236 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:42:12.278086 (+ 1850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:42:12.278092 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:42:12.278125 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:42:12.278629 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:12.278643 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:12.281106 (+ 2463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:12.281114 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:12.289929 (+ 8815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:12.289943 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:12.293136 (+ 3193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:12.293143 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:12.293318 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:12.293352 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 10:42:12.293435 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 10:42:12.293572 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18871,"mutex_wait_us":226,"thread_start_us":22941,"threads_started":1}
I20260925 10:42:12.294817 4856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:12.257255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49410 (local address 127.23.100.2:46423)
0925 10:42:12.275554 (+ 18299us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:12.275567 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:12.276869 (+ 1302us) server_negotiation.cc:407] Connection header received
0925 10:42:12.277003 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:12.277014 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:12.277335 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:12.277556 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:42:12.279371 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:12.280926 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:12.290123 (+ 9197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:12.292663 (+ 2540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:12.292804 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:12.292985 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:12.294333 (+ 1348us) server_negotiation.cc:299] Negotiation successful
0925 10:42:12.294508 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18079,"thread_start_us":21068,"threads_started":1}
I20260925 10:42:12.295879 4773 tablet_service.cc:1467] Processing CreateTablet for tablet d097b0bc2adb4ae1a4c99b93088c945c (DEFAULT_TABLE table=test-table [id=05ffe3f96b2b45659569bb7f761016cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:42:12.297538 4773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d097b0bc2adb4ae1a4c99b93088c945c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:42:12.298532 4635 tablet_service.cc:1467] Processing CreateTablet for tablet d097b0bc2adb4ae1a4c99b93088c945c (DEFAULT_TABLE table=test-table [id=05ffe3f96b2b45659569bb7f761016cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:42:12.299875 4635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d097b0bc2adb4ae1a4c99b93088c945c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:42:12.305665 4494 tablet_service.cc:1467] Processing CreateTablet for tablet d097b0bc2adb4ae1a4c99b93088c945c (DEFAULT_TABLE table=test-table [id=05ffe3f96b2b45659569bb7f761016cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:42:12.307331 4494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d097b0bc2adb4ae1a4c99b93088c945c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:42:12.329849 4859 tablet_bootstrap.cc:492] T d097b0bc2adb4ae1a4c99b93088c945c P 42798400926640c6a4caf15907941d2f: Bootstrap starting.
I20260925 10:42:12.336817 4860 tablet_bootstrap.cc:492] T d097b0bc2adb4ae1a4c99b93088c945c P ac3d0d1927d24613bddec74a922e7bf7: Bootstrap starting.
I20260925 10:42:12.337569 4859 tablet_bootstrap.cc:654] T d097b0bc2adb4ae1a4c99b93088c945c P 42798400926640c6a4caf15907941d2f: Neither blocks nor log segments found. Creating new log.
I20260925 10:42:12.339054 4859 log.cc:824] T d097b0bc2adb4ae1a4c99b93088c945c P 42798400926640c6a4caf15907941d2f: Log is configured to *not* fsync() on all Append() calls
I20260925 10:42:12.339156 4861 tablet_bootstrap.cc:492] T d097b0bc2adb4ae1a4c99b93088c945c P 1b102e5f357d4ef49c5c6703954d5b06: Bootstrap starting.
I20260925 10:42:12.340593 4860 tablet_bootstrap.cc:654] T d097b0bc2adb4ae1a4c99b93088c945c P ac3d0d1927d24613bddec74a922e7bf7: Neither blocks nor log segments found. Creating new log.
I20260925 10:42:12.342165 4860 log.cc:824] T d097b0bc2adb4ae1a4c99b93088c945c P ac3d0d1927d24613bddec74a922e7bf7: Log is configured to *not* fsync() on all Append() calls
I20260925 10:42:12.343694 4861 tablet_bootstrap.cc:654] T d097b0bc2adb4ae1a4c99b93088c945c P 1b102e5f357d4ef49c5c6703954d5b06: Neither blocks nor log segments found. Creating new log.
I20260925 10:42:12.344780 4861 log.cc:824] T d097b0bc2adb4ae1a4c99b93088c945c P 1b102e5f357d4ef49c5c6703954d5b06: Log is configured to *not* fsync() on all Append() calls
I20260925 10:42:12.350917 4859 tablet_bootstrap.cc:492] T d097b0bc2adb4ae1a4c99b93088c945c P 42798400926640c6a4caf15907941d2f: No bootstrap required, opened a new log
I20260925 10:42:12.351271 4859 ts_tablet_manager.cc:1397] T d097b0bc2adb4ae1a4c99b93088c945c P 42798400926640c6a4caf15907941d2f: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.003s
I20260925 10:42:12.351620 4861 tablet_bootstrap.cc:492] T d097b0bc2adb4ae1a4c99b93088c945c P 1b102e5f357d4ef49c5c6703954d5b06: No bootstrap required, opened a new log
I20260925 10:42:12.351872 4861 ts_tablet_manager.cc:1397] T d097b0bc2adb4ae1a4c99b93088c945c P 1b102e5f357d4ef49c5c6703954d5b06: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.004s
I20260925 10:42:12.361981 4859 raft_consensus.cc:348] T d097b0bc2adb4ae1a4c99b93088c945c P 42798400926640c6a4caf15907941d2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b102e5f357d4ef49c5c6703954d5b06" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37795 } } peers { permanent_uuid: "ac3d0d1927d24613bddec74a922e7bf7" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 46423 } } peers { permanent_uuid: "42798400926640c6a4caf15907941d2f" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 42537 } }
I20260925 10:42:12.365010 4860 tablet_bootstrap.cc:492] T d097b0bc2adb4ae1a4c99b93088c945c P ac3d0d1927d24613bddec74a922e7bf7: No bootstrap required, opened a new log
I20260925 10:42:12.365337 4860 ts_tablet_manager.cc:1397] T d097b0bc2adb4ae1a4c99b93088c945c P ac3d0d1927d24613bddec74a922e7bf7: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.014s
I20260925 10:42:12.366496 4861 raft_consensus.cc:348] T d097b0bc2adb4ae1a4c99b93088c945c P 1b102e5f357d4ef49c5c6703954d5b06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b102e5f357d4ef49c5c6703954d5b06" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37795 } } peers { permanent_uuid: "ac3d0d1927d24613bddec74a922e7bf7" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 46423 } } peers { permanent_uuid: "42798400926640c6a4caf15907941d2f" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 42537 } }
I20260925 10:42:12.366977 4861 raft_consensus.cc:374] T d097b0bc2adb4ae1a4c99b93088c945c P 1b102e5f357d4ef49c5c6703954d5b06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:42:12.367125 4861 raft_consensus.cc:729] T d097b0bc2adb4ae1a4c99b93088c945c P 1b102e5f357d4ef49c5c6703954d5b06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b102e5f357d4ef49c5c6703954d5b06, State: Initialized, Role: FOLLOWER
I20260925 10:42:12.367923 4861 consensus_queue.cc:260] T d097b0bc2adb4ae1a4c99b93088c945c P 1b102e5f357d4ef49c5c6703954d5b06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b102e5f357d4ef49c5c6703954d5b06" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37795 } } peers { permanent_uuid: "ac3d0d1927d24613bddec74a922e7bf7" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 46423 } } peers { permanent_uuid: "42798400926640c6a4caf15907941d2f" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 42537 } }
I20260925 10:42:12.373093 4859 raft_consensus.cc:374] T d097b0bc2adb4ae1a4c99b93088c945c P 42798400926640c6a4caf15907941d2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:42:12.373214 4859 raft_consensus.cc:729] T d097b0bc2adb4ae1a4c99b93088c945c P 42798400926640c6a4caf15907941d2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42798400926640c6a4caf15907941d2f, State: Initialized, Role: FOLLOWER
I20260925 10:42:12.373867 4859 consensus_queue.cc:260] T d097b0bc2adb4ae1a4c99b93088c945c P 42798400926640c6a4caf15907941d2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b102e5f357d4ef49c5c6703954d5b06" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37795 } } peers { permanent_uuid: "ac3d0d1927d24613bddec74a922e7bf7" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 46423 } } peers { permanent_uuid: "42798400926640c6a4caf15907941d2f" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 42537 } }
I20260925 10:42:12.376374 4861 ts_tablet_manager.cc:1428] T d097b0bc2adb4ae1a4c99b93088c945c P 1b102e5f357d4ef49c5c6703954d5b06: Time spent starting tablet: real 0.024s user 0.016s sys 0.003s
I20260925 10:42:12.376112 4860 raft_consensus.cc:348] T d097b0bc2adb4ae1a4c99b93088c945c P ac3d0d1927d24613bddec74a922e7bf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b102e5f357d4ef49c5c6703954d5b06" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37795 } } peers { permanent_uuid: "ac3d0d1927d24613bddec74a922e7bf7" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 46423 } } peers { permanent_uuid: "42798400926640c6a4caf15907941d2f" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 42537 } }
I20260925 10:42:12.376511 4860 raft_consensus.cc:374] T d097b0bc2adb4ae1a4c99b93088c945c P ac3d0d1927d24613bddec74a922e7bf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:42:12.376621 4860 raft_consensus.cc:729] T d097b0bc2adb4ae1a4c99b93088c945c P ac3d0d1927d24613bddec74a922e7bf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac3d0d1927d24613bddec74a922e7bf7, State: Initialized, Role: FOLLOWER
I20260925 10:42:12.376834 4559 heartbeater.cc:503] Master 127.23.100.62:46161 was elected leader, sending a full tablet report...
I20260925 10:42:12.377197 4860 consensus_queue.cc:260] T d097b0bc2adb4ae1a4c99b93088c945c P ac3d0d1927d24613bddec74a922e7bf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b102e5f357d4ef49c5c6703954d5b06" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37795 } } peers { permanent_uuid: "ac3d0d1927d24613bddec74a922e7bf7" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 46423 } } peers { permanent_uuid: "42798400926640c6a4caf15907941d2f" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 42537 } }
I20260925 10:42:12.384222 4700 heartbeater.cc:503] Master 127.23.100.62:46161 was elected leader, sending a full tablet report...
I20260925 10:42:12.384909 4838 heartbeater.cc:503] Master 127.23.100.62:46161 was elected leader, sending a full tablet report...
I20260925 10:42:12.390426 4859 ts_tablet_manager.cc:1428] T d097b0bc2adb4ae1a4c99b93088c945c P 42798400926640c6a4caf15907941d2f: Time spent starting tablet: real 0.039s user 0.014s sys 0.000s
I20260925 10:42:12.391053 4866 raft_consensus.cc:482] T d097b0bc2adb4ae1a4c99b93088c945c P 1b102e5f357d4ef49c5c6703954d5b06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:42:12.391321 4866 raft_consensus.cc:504] T d097b0bc2adb4ae1a4c99b93088c945c P 1b102e5f357d4ef49c5c6703954d5b06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b102e5f357d4ef49c5c6703954d5b06" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37795 } } peers { permanent_uuid: "ac3d0d1927d24613bddec74a922e7bf7" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 46423 } } peers { permanent_uuid: "42798400926640c6a4caf15907941d2f" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 42537 } }
I20260925 10:42:12.394045 4866 leader_election.cc:288] T d097b0bc2adb4ae1a4c99b93088c945c P 1b102e5f357d4ef49c5c6703954d5b06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac3d0d1927d24613bddec74a922e7bf7 (127.23.100.2:46423), 42798400926640c6a4caf15907941d2f (127.23.100.3:42537)
I20260925 10:42:12.394258 4860 ts_tablet_manager.cc:1428] T d097b0bc2adb4ae1a4c99b93088c945c P ac3d0d1927d24613bddec74a922e7bf7: Time spent starting tablet: real 0.029s user 0.011s sys 0.001s
Sep 25 10:42:12 dist-test-slave-dxdd krb5kdc[3352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332931, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.62@KRBTEST.COM for HTTP/127.23.100.20@KRBTEST.COM
I20260925 10:42:12.406553 4858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:12.393749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:50379 (local address 127.23.100.3:42537)
0925 10:42:12.393908 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:12.393914 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:12.396928 (+ 3014us) server_negotiation.cc:407] Connection header received
0925 10:42:12.397217 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:12.397222 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:12.397281 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:12.397333 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:42:12.398550 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:12.399608 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:12.403121 (+ 3513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:12.404292 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:12.404409 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:12.404483 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:12.406160 (+ 1677us) server_negotiation.cc:299] Negotiation successful
0925 10:42:12.406285 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 10:42:12.407358 4869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:12.393750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.3:42537 (local address 127.23.100.1:50379)
0925 10:42:12.396842 (+ 3092us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:12.396860 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:42:12.397079 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:42:12.397904 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:42:12.397909 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:42:12.397933 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:42:12.398378 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:12.398391 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:12.399780 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:12.399785 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:12.402946 (+ 3161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:12.402962 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:12.405814 (+ 2852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:12.405819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:12.405981 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:12.406010 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 10:42:12.406922 (+ 912us) client_negotiation.cc:240] Negotiation successful
0925 10:42:12.407052 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":985,"thread_start_us":804,"threads_started":1}
I20260925 10:42:12.408434 4793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d097b0bc2adb4ae1a4c99b93088c945c" candidate_uuid: "1b102e5f357d4ef49c5c6703954d5b06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42798400926640c6a4caf15907941d2f" is_pre_election: true
I20260925 10:42:12.408958 4793 raft_consensus.cc:2454] T d097b0bc2adb4ae1a4c99b93088c945c P 42798400926640c6a4caf15907941d2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b102e5f357d4ef49c5c6703954d5b06 in term 0.
I20260925 10:42:12.409682 4450 leader_election.cc:302] T d097b0bc2adb4ae1a4c99b93088c945c P 1b102e5f357d4ef49c5c6703954d5b06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b102e5f357d4ef49c5c6703954d5b06, 42798400926640c6a4caf15907941d2f; no voters:
I20260925 10:42:12.410127 4866 raft_consensus.cc:2781] T d097b0bc2adb4ae1a4c99b93088c945c P 1b102e5f357d4ef49c5c6703954d5b06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:42:12.412561 4856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:12.394675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:49293 (local address 127.23.100.2:46423)
0925 10:42:12.394851 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:12.394856 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:12.402606 (+ 7750us) server_negotiation.cc:407] Connection header received
0925 10:42:12.402852 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:12.402858 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:12.402941 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:12.403003 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:42:12.406839 (+ 3836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:12.407946 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:12.410114 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:12.411346 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:12.411476 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:12.411557 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:12.412185 (+ 628us) server_negotiation.cc:299] Negotiation successful
0925 10:42:12.412315 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 10:42:12.413271 4870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:12.395018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.2:46423 (local address 127.23.100.1:49293)
0925 10:42:12.402504 (+ 7486us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:12.402525 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 10:42:12.402701 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:42:12.403193 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:42:12.403197 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:42:12.403220 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:42:12.406634 (+ 3414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:12.406649 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:12.408129 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:12.408134 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:12.409933 (+ 1799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:12.409946 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:12.411787 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:12.411794 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:12.411994 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:12.412028 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 10:42:12.412719 (+ 691us) client_negotiation.cc:240] Negotiation successful
0925 10:42:12.412957 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7327,"thread_start_us":7127,"threads_started":1}
I20260925 10:42:12.410266 4866 raft_consensus.cc:482] T d097b0bc2adb4ae1a4c99b93088c945c P 1b102e5f357d4ef49c5c6703954d5b06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:42:12.414446 4866 raft_consensus.cc:3037] T d097b0bc2adb4ae1a4c99b93088c945c P 1b102e5f357d4ef49c5c6703954d5b06 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:42:12.414386 4655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d097b0bc2adb4ae1a4c99b93088c945c" candidate_uuid: "1b102e5f357d4ef49c5c6703954d5b06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac3d0d1927d24613bddec74a922e7bf7" is_pre_election: true
I20260925 10:42:12.414896 4655 raft_consensus.cc:2454] T d097b0bc2adb4ae1a4c99b93088c945c P ac3d0d1927d24613bddec74a922e7bf7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b102e5f357d4ef49c5c6703954d5b06 in term 0.
I20260925 10:42:12.417718 4866 raft_consensus.cc:504] T d097b0bc2adb4ae1a4c99b93088c945c P 1b102e5f357d4ef49c5c6703954d5b06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b102e5f357d4ef49c5c6703954d5b06" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37795 } } peers { permanent_uuid: "ac3d0d1927d24613bddec74a922e7bf7" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 46423 } } peers { permanent_uuid: "42798400926640c6a4caf15907941d2f" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 42537 } }
I20260925 10:42:12.418467 4866 leader_election.cc:288] T d097b0bc2adb4ae1a4c99b93088c945c P 1b102e5f357d4ef49c5c6703954d5b06 [CANDIDATE]: Term 1 election: Requested vote from peers ac3d0d1927d24613bddec74a922e7bf7 (127.23.100.2:46423), 42798400926640c6a4caf15907941d2f (127.23.100.3:42537)
I20260925 10:42:12.418994 4655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d097b0bc2adb4ae1a4c99b93088c945c" candidate_uuid: "1b102e5f357d4ef49c5c6703954d5b06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac3d0d1927d24613bddec74a922e7bf7"
I20260925 10:42:12.419183 4655 raft_consensus.cc:3037] T d097b0bc2adb4ae1a4c99b93088c945c P ac3d0d1927d24613bddec74a922e7bf7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:42:12.419183 4793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d097b0bc2adb4ae1a4c99b93088c945c" candidate_uuid: "1b102e5f357d4ef49c5c6703954d5b06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42798400926640c6a4caf15907941d2f"
I20260925 10:42:12.419385 4793 raft_consensus.cc:3037] T d097b0bc2adb4ae1a4c99b93088c945c P 42798400926640c6a4caf15907941d2f [term 0 FOLLOWER]: Advancing to term 1
W20260925 10:42:12.419864 4839 tablet.cc:2369] T d097b0bc2adb4ae1a4c99b93088c945c P 42798400926640c6a4caf15907941d2f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:42:12.422762 4793 raft_consensus.cc:2454] T d097b0bc2adb4ae1a4c99b93088c945c P 42798400926640c6a4caf15907941d2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b102e5f357d4ef49c5c6703954d5b06 in term 1.
I20260925 10:42:12.423005 4655 raft_consensus.cc:2454] T d097b0bc2adb4ae1a4c99b93088c945c P ac3d0d1927d24613bddec74a922e7bf7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b102e5f357d4ef49c5c6703954d5b06 in term 1.
I20260925 10:42:12.423290 4450 leader_election.cc:302] T d097b0bc2adb4ae1a4c99b93088c945c P 1b102e5f357d4ef49c5c6703954d5b06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b102e5f357d4ef49c5c6703954d5b06, 42798400926640c6a4caf15907941d2f; no voters:
I20260925 10:42:12.423753 4866 raft_consensus.cc:2781] T d097b0bc2adb4ae1a4c99b93088c945c P 1b102e5f357d4ef49c5c6703954d5b06 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:42:12.424909 4866 raft_consensus.cc:686] T d097b0bc2adb4ae1a4c99b93088c945c P 1b102e5f357d4ef49c5c6703954d5b06 [term 1 LEADER]: Becoming Leader. State: Replica: 1b102e5f357d4ef49c5c6703954d5b06, State: Running, Role: LEADER
I20260925 10:42:12.425705 4866 consensus_queue.cc:237] T d097b0bc2adb4ae1a4c99b93088c945c P 1b102e5f357d4ef49c5c6703954d5b06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b102e5f357d4ef49c5c6703954d5b06" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37795 } } peers { permanent_uuid: "ac3d0d1927d24613bddec74a922e7bf7" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 46423 } } peers { permanent_uuid: "42798400926640c6a4caf15907941d2f" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 42537 } }
I20260925 10:42:12.439522 4363 catalog_manager.cc:5495] T d097b0bc2adb4ae1a4c99b93088c945c P 1b102e5f357d4ef49c5c6703954d5b06 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b102e5f357d4ef49c5c6703954d5b06 (127.23.100.1). New cstate: current_term: 1 leader_uuid: "1b102e5f357d4ef49c5c6703954d5b06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b102e5f357d4ef49c5c6703954d5b06" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac3d0d1927d24613bddec74a922e7bf7" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 46423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42798400926640c6a4caf15907941d2f" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 42537 } health_report { overall_health: UNKNOWN } } }
I20260925 10:42:12.519536 4854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:12.511818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53604 (local address 127.23.100.1:37795)
0925 10:42:12.512100 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:12.512106 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:12.512128 (+ 22us) server_negotiation.cc:407] Connection header received
0925 10:42:12.512243 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:12.512250 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:12.512380 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:12.512560 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:42:12.513173 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:12.514297 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:12.516338 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:12.516961 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:12.517081 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:12.517792 (+ 711us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:42:12.518535 (+ 743us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:42:12.518671 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:12.519134 (+ 463us) server_negotiation.cc:299] Negotiation successful
0925 10:42:12.519217 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260925 10:42:12.535494 4655 raft_consensus.cc:1261] T d097b0bc2adb4ae1a4c99b93088c945c P ac3d0d1927d24613bddec74a922e7bf7 [term 1 FOLLOWER]: Refusing update from remote peer 1b102e5f357d4ef49c5c6703954d5b06: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:42:12.535791 4793 raft_consensus.cc:1261] T d097b0bc2adb4ae1a4c99b93088c945c P 42798400926640c6a4caf15907941d2f [term 1 FOLLOWER]: Refusing update from remote peer 1b102e5f357d4ef49c5c6703954d5b06: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:42:12.536501 4866 consensus_queue.cc:1035] T d097b0bc2adb4ae1a4c99b93088c945c P 1b102e5f357d4ef49c5c6703954d5b06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac3d0d1927d24613bddec74a922e7bf7" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 46423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:42:12.536839 4871 consensus_queue.cc:1035] T d097b0bc2adb4ae1a4c99b93088c945c P 1b102e5f357d4ef49c5c6703954d5b06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42798400926640c6a4caf15907941d2f" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 42537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:42:12.570055 4874 mvcc.cc:205] Tried to move back new op lower bound from 7333203691652825088 to 7333203691220754432. Current Snapshot: MvccSnapshot[applied={T|T < 7333203691652825088}]
I20260925 10:42:12.572104 4875 mvcc.cc:205] Tried to move back new op lower bound from 7333203691652825088 to 7333203691220754432. Current Snapshot: MvccSnapshot[applied={T|T < 7333203691652825088}]
I20260925 10:42:12.595543 4876 mvcc.cc:205] Tried to move back new op lower bound from 7333203691652825088 to 7333203691220754432. Current Snapshot: MvccSnapshot[applied={T|T < 7333203691652825088}]
I20260925 10:42:12.602632 4363 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39702:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:42:12.602909 4363 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39702:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:42:12.608104 4363 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e8f9b2e6213b4dcba485041b6110ee68: Sending DeleteTablet for 3 replicas of tablet d097b0bc2adb4ae1a4c99b93088c945c
I20260925 10:42:12.609962 4494 tablet_service.cc:1514] Processing DeleteTablet for tablet d097b0bc2adb4ae1a4c99b93088c945c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:42:12 UTC) from {username='kudu', principal='kudu/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:53598
I20260925 10:42:12.610237 4773 tablet_service.cc:1514] Processing DeleteTablet for tablet d097b0bc2adb4ae1a4c99b93088c945c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:42:12 UTC) from {username='kudu', principal='kudu/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:47784
I20260925 10:42:12.610584 4635 tablet_service.cc:1514] Processing DeleteTablet for tablet d097b0bc2adb4ae1a4c99b93088c945c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:42:12 UTC) from {username='kudu', principal='kudu/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:49410
I20260925 10:42:12.616130 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 4400
I20260925 10:42:12.625016 4884 tablet_replica.cc:330] T d097b0bc2adb4ae1a4c99b93088c945c P 42798400926640c6a4caf15907941d2f: stopping tablet replica
I20260925 10:42:12.625622 4885 tablet_replica.cc:330] T d097b0bc2adb4ae1a4c99b93088c945c P ac3d0d1927d24613bddec74a922e7bf7: stopping tablet replica
I20260925 10:42:12.626540 4885 raft_consensus.cc:2229] T d097b0bc2adb4ae1a4c99b93088c945c P ac3d0d1927d24613bddec74a922e7bf7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:42:12.627101 4884 raft_consensus.cc:2229] T d097b0bc2adb4ae1a4c99b93088c945c P 42798400926640c6a4caf15907941d2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:42:12.627892 4885 raft_consensus.cc:2258] T d097b0bc2adb4ae1a4c99b93088c945c P ac3d0d1927d24613bddec74a922e7bf7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:42:12.635306 4884 raft_consensus.cc:2258] T d097b0bc2adb4ae1a4c99b93088c945c P 42798400926640c6a4caf15907941d2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:42:12.639746 4885 ts_tablet_manager.cc:1905] T d097b0bc2adb4ae1a4c99b93088c945c P ac3d0d1927d24613bddec74a922e7bf7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:42:12.641996 4854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:12.617584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53608 (local address 127.23.100.1:37795)
0925 10:42:12.617786 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:12.617794 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:12.617815 (+ 21us) server_negotiation.cc:407] Connection header received
0925 10:42:12.617878 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:12.617886 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:12.617969 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:12.618071 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:42:12.621066 (+ 2995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:12.624143 (+ 3077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:12.626727 (+ 2584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:12.627753 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:12.627904 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:12.632701 (+ 4797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:42:12.632713 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:42:12.632731 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:42:12.632779 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:42:12.637154 (+ 4375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:12.638421 (+ 1267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:12.638430 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:12.638433 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:12.638630 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:12.639963 (+ 1333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:12.639969 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:12.639973 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:12.640395 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:42:12.640684 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:12.641355 (+ 671us) server_negotiation.cc:299] Negotiation successful
0925 10:42:12.641587 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 10:42:12.642769 4554 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:42:12.643818 4884 ts_tablet_manager.cc:1905] T d097b0bc2adb4ae1a4c99b93088c945c P 42798400926640c6a4caf15907941d2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:42:12.634935 4886 tablet_replica.cc:330] T d097b0bc2adb4ae1a4c99b93088c945c P 1b102e5f357d4ef49c5c6703954d5b06: stopping tablet replica
I20260925 10:42:12.651019 4884 ts_tablet_manager.cc:1918] T d097b0bc2adb4ae1a4c99b93088c945c P 42798400926640c6a4caf15907941d2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:42:12.651238 4884 log.cc:1196] T d097b0bc2adb4ae1a4c99b93088c945c P 42798400926640c6a4caf15907941d2f: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-2/wal/wals/d097b0bc2adb4ae1a4c99b93088c945c
I20260925 10:42:12.651890 4884 ts_tablet_manager.cc:1939] T d097b0bc2adb4ae1a4c99b93088c945c P 42798400926640c6a4caf15907941d2f: Deleting consensus metadata
I20260925 10:42:12.654060 4350 catalog_manager.cc:4841] TS 42798400926640c6a4caf15907941d2f (127.23.100.3:42537): tablet d097b0bc2adb4ae1a4c99b93088c945c (table test-table [id=05ffe3f96b2b45659569bb7f761016cd]) successfully deleted
I20260925 10:42:12.656757 4885 ts_tablet_manager.cc:1918] T d097b0bc2adb4ae1a4c99b93088c945c P ac3d0d1927d24613bddec74a922e7bf7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:42:12.656970 4885 log.cc:1196] T d097b0bc2adb4ae1a4c99b93088c945c P ac3d0d1927d24613bddec74a922e7bf7: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790332740062128-23952-0/minicluster-data/ts-1/wal/wals/d097b0bc2adb4ae1a4c99b93088c945c
I20260925 10:42:12.657521 4885 ts_tablet_manager.cc:1939] T d097b0bc2adb4ae1a4c99b93088c945c P ac3d0d1927d24613bddec74a922e7bf7: Deleting consensus metadata
I20260925 10:42:12.659343 4349 catalog_manager.cc:4841] TS ac3d0d1927d24613bddec74a922e7bf7 (127.23.100.2:46423): tablet d097b0bc2adb4ae1a4c99b93088c945c (table test-table [id=05ffe3f96b2b45659569bb7f761016cd]) successfully deleted
I20260925 10:42:12.650288 4886 raft_consensus.cc:2229] T d097b0bc2adb4ae1a4c99b93088c945c P 1b102e5f357d4ef49c5c6703954d5b06 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:42:12.748347 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 4400
I20260925 10:42:12.748339 4886 raft_consensus.cc:2258] T d097b0bc2adb4ae1a4c99b93088c945c P 1b102e5f357d4ef49c5c6703954d5b06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:42:12.765268 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 4563
W20260925 10:42:12.765434 4349 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.100.1:37795 (error 108)
I20260925 10:42:12.767879 4855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:12.767287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:37795 (local address 127.0.0.1:53622)
0925 10:42:12.767599 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:12.767715 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.1:37795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208}
W20260925 10:42:12.768294 4349 catalog_manager.cc:4584] TS 1b102e5f357d4ef49c5c6703954d5b06 (127.23.100.1:37795): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d097b0bc2adb4ae1a4c99b93088c945c: Network error: Client connection negotiation failed: client connection to 127.23.100.1:37795: connect: Connection refused (error 111)
I20260925 10:42:12.785988 4856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:12.767281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49426 (local address 127.23.100.2:46423)
0925 10:42:12.767443 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:12.767449 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:12.767466 (+ 17us) server_negotiation.cc:407] Connection header received
0925 10:42:12.767540 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:12.767545 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:12.767618 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:12.767718 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:42:12.769378 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:12.770610 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:12.771867 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:12.772443 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:12.772565 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:12.779511 (+ 6946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:42:12.779532 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:42:12.779553 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:42:12.779600 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:42:12.783310 (+ 3710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:12.783966 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:12.783976 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:12.783979 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:12.784145 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:12.784462 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:12.784467 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:12.784470 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:12.784921 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:42:12.785200 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:12.785410 (+ 210us) server_negotiation.cc:299] Negotiation successful
0925 10:42:12.785661 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 10:42:12.786793 4695 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:42:12.822279 4855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:12.821713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:37795 (local address 127.0.0.1:53636)
0925 10:42:12.821984 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:12.822115 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.1:37795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
I20260925 10:42:12.823432 4855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:12.823049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:37795 (local address 127.0.0.1:53640)
0925 10:42:12.823224 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:12.823293 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.1:37795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260925 10:42:12.882500 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 4563
I20260925 10:42:12.884929 4855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:12.884255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:37795 (local address 127.0.0.1:53654)
0925 10:42:12.884601 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:12.884721 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.1:37795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
I20260925 10:42:12.886122 4855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:12.885716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:37795 (local address 127.0.0.1:53658)
0925 10:42:12.885875 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:12.885971 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.1:37795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 10:42:12.899600 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 4706
I20260925 10:42:12.914891 4858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:12.900404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47794 (local address 127.23.100.3:42537)
0925 10:42:12.900612 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:12.900616 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:12.900634 (+ 18us) server_negotiation.cc:407] Connection header received
0925 10:42:12.900787 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:12.900791 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:12.900862 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:12.900957 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:42:12.902640 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:12.903360 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:12.905184 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:12.905559 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:12.905651 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:12.908909 (+ 3258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:42:12.908923 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:42:12.908947 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:42:12.908980 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:42:12.911784 (+ 2804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:12.912608 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:12.912615 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:12.912618 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:12.912744 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:12.913135 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:12.913138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:12.913140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:12.913487 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:42:12.913702 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:12.914147 (+ 445us) server_negotiation.cc:299] Negotiation successful
0925 10:42:12.914342 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260925 10:42:12.915575 4833 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:42:12.966980 4855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:12.966387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:37795 (local address 127.0.0.1:53674)
0925 10:42:12.966630 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:12.966743 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.1:37795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260925 10:42:12.968166 4855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:12.967734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:37795 (local address 127.0.0.1:53682)
0925 10:42:12.967953 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:12.968036 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.1:37795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260925 10:42:13.013522 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 4706
I20260925 10:42:13.030550 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 4330
I20260925 10:42:13.031689 4393 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:42:13.143361 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 4330
I20260925 10:42:13.167129 23952 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 10:42:13.502619 23952 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 10:42:13.502840 23952 mini_ranger.cc:66] Stopping Ranger...
I20260925 10:42:13.848336 23952 mini_ranger.cc:68] Stopped Ranger
2026-09-25 10:42:13.848 UTC [3382] LOG: received smart shutdown request
2026-09-25 10:42:13.851 UTC [3382] LOG: background worker "logical replication launcher" (PID 3391) exited with exit code 1
2026-09-25 10:42:13.851 UTC [3386] LOG: shutting down
2026-09-25 10:42:14.006 UTC [3382] LOG: database system is shut down
2026-09-25T10:42:14Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49668 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:42:14 dist-test-slave-dxdd krb5kdc[4907](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:42:14 dist-test-slave-dxdd krb5kdc[4907](info): set up 2 sockets
Sep 25 10:42:14 dist-test-slave-dxdd krb5kdc[4907](info): commencing operation
krb5kdc: starting...
W20260925 10:42:15.214653 23952 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Sep 25 10:42:15 dist-test-slave-dxdd krb5kdc[4907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:42:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:42:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
I20260925 10:42:15.439767 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:37805
--webserver_interface=127.23.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:33749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.62:37805,127.23.100.61:37529,127.23.100.60:33377
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:42:15.756516 4920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:42:15.756800 4920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:42:15.756909 4920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:42:15.763974 4920 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:42:15.764077 4920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:42:15.764125 4920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:42:15.764202 4920 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:42:15.764278 4920 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:42:15.774701 4920 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:33749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.62:37805,127.23.100.61:37529,127.23.100.60:33377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:37805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:42:15.776001 4920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:42:15.777271 4920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:42:15.788508 4925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:42:15.788570 4926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:42:15.788604 4928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:42:15.789623 4920 server_base.cc:1034] running on GCE node
I20260925 10:42:15.790378 4920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:42:15.791637 4920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:42:15.792850 4920 hybrid_clock.cc:648] HybridClock initialized: now 1790332935792817 us; error 52 us; skew 500 ppm
I20260925 10:42:15.793181 4920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:42:15 dist-test-slave-dxdd krb5kdc[4907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332935, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:42:15.796202 4920 init.cc:377] Logged in from keytab as kudu/127.23.100.62@KRBTEST.COM (short username kudu)
I20260925 10:42:15.797523 4920 webserver.cc:460] Webserver started at http://127.23.100.62:45097/ using document root <none> and password file <none>
I20260925 10:42:15.798172 4920 fs_manager.cc:359] Metadata directory not provided
I20260925 10:42:15.798336 4920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:42:15.798640 4920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:42:15.801401 4920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/minicluster-data/master-0/data/instance:
uuid: "355e9e0b4d5e409596876e18547a020b"
format_stamp: "Formatted at 2026-09-25 10:42:15 on dist-test-slave-dxdd"
server_key: "1f0bb8ec81deeac2c81e66fe9dfbdc9a"
server_key_iv: "40d40ef6953714f8404bc8b068242e40"
server_key_version: "encryptionkey@0"
I20260925 10:42:15.802158 4920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/minicluster-data/master-0/wal/instance:
uuid: "355e9e0b4d5e409596876e18547a020b"
format_stamp: "Formatted at 2026-09-25 10:42:15 on dist-test-slave-dxdd"
server_key: "1f0bb8ec81deeac2c81e66fe9dfbdc9a"
server_key_iv: "40d40ef6953714f8404bc8b068242e40"
server_key_version: "encryptionkey@0"
I20260925 10:42:15.807008 4920 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 10:42:15.810117 4935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:42:15.811357 4920 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 10:42:15.811525 4920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/minicluster-data/master-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/minicluster-data/master-0/wal
uuid: "355e9e0b4d5e409596876e18547a020b"
format_stamp: "Formatted at 2026-09-25 10:42:15 on dist-test-slave-dxdd"
server_key: "1f0bb8ec81deeac2c81e66fe9dfbdc9a"
server_key_iv: "40d40ef6953714f8404bc8b068242e40"
server_key_version: "encryptionkey@0"
I20260925 10:42:15.811714 4920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:42:15.826228 4920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:42:15.830893 4920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:42:15.831256 4920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:42:15.848675 4920 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.62:37805
I20260925 10:42:15.848706 4986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.62:37805 every 8 connection(s)
I20260925 10:42:15.849994 4920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
I20260925 10:42:15.853533 4987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:42:15.860278 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 4920
I20260925 10:42:15.860502 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/minicluster-data/master-0/wal/instance
I20260925 10:42:15.859683 4987 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.62" port: 37805 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:42:15.861331 23952 external_mini_cluster.cc:1428] Setting key 352192c6abf4c0e8e2344cd4b7d1f6b0
Sep 25 10:42:15 dist-test-slave-dxdd krb5kdc[4907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332935, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.62@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.100.61@KRBTEST.COM; defaulting to no policy
I20260925 10:42:15.885566 4990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:15.863290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33140 (local address 127.23.100.62:37805)
0925 10:42:15.869562 (+ 6272us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:15.869568 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:15.869596 (+ 28us) server_negotiation.cc:407] Connection header received
0925 10:42:15.869714 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:15.869719 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:15.869920 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:15.870128 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:42:15.871401 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:15.872568 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:15.873916 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:15.874536 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:15.874722 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:15.877980 (+ 3258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:42:15.877987 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:42:15.877997 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:42:15.878022 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:42:15.881734 (+ 3712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:15.882442 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:15.882452 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:15.882457 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:15.882662 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:15.883030 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:15.883035 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:15.883038 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:15.883687 (+ 649us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:42:15.883843 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:15.884317 (+ 474us) server_negotiation.cc:299] Negotiation successful
0925 10:42:15.884705 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":167,"server-negotiator.queue_time_us":6115,"spinlock_wait_cycles":64384,"thread_start_us":5931,"threads_started":1}
I20260925 10:42:15.885594 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:15.863290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:37805 (local address 127.0.0.1:33140)
0925 10:42:15.864157 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:15.864204 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 10:42:15.867088 (+ 2884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:42:15.870135 (+ 3047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:42:15.870141 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:42:15.870853 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:42:15.871170 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:15.871185 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:15.872736 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:15.872743 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:15.873785 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:15.873796 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:15.874754 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:15.874761 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:15.874930 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:15.875535 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:42:15.875557 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:42:15.877782 (+ 2225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:42:15.881861 (+ 4079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:42:15.881870 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:42:15.881878 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:42:15.882261 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:42:15.882772 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:42:15.882775 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:42:15.882777 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:42:15.882908 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:42:15.883808 (+ 900us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:42:15.883814 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:42:15.883988 (+ 174us) client_negotiation.cc:769] Sending connection context
0925 10:42:15.884189 (+ 201us) client_negotiation.cc:240] Negotiation successful
0925 10:42:15.884562 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":371,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.61@KRBTEST.COM" created.
I20260925 10:42:15.888409 4987 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.61" port: 37529 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:42:15.889525 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:15.889072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.61:37529 (local address 127.0.0.1:57164)
0925 10:42:15.889255 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:15.889382 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.61:37529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 10:42:15.889804 4937 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.100.61:37529: connect: Connection refused (error 111)
I20260925 10:42:15.891418 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:15.890994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.61:37529 (local address 127.0.0.1:57166)
0925 10:42:15.891167 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:15.891259 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.61:37529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 10:42:15.891639 4987 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.61:37529: Network error: Client connection negotiation failed: client connection to 127.23.100.61:37529: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.61.keytab.
Entry for principal kudu/127.23.100.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.61.keytab.
I20260925 10:42:15.912940 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.61:37529
--webserver_interface=127.23.100.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.61.keytab
--principal=kudu/127.23.100.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:33749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.62:37805,127.23.100.61:37529,127.23.100.60:33377
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:42:15.940960 4987 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.61" port: 37529 } attempt: 1
I20260925 10:42:15.942119 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:15.941684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.61:37529 (local address 127.0.0.1:57174)
0925 10:42:15.941874 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:15.941980 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.61:37529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 10:42:15.943159 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:15.942865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.61:37529 (local address 127.0.0.1:57178)
0925 10:42:15.942978 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:15.943043 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.61:37529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 10:42:15.943352 4987 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.61:37529: Network error: Client connection negotiation failed: client connection to 127.23.100.61:37529: connect: Connection refused (error 111)
I20260925 10:42:16.011670 4987 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.61" port: 37529 } attempt: 2
I20260925 10:42:16.012894 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:16.012426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.61:37529 (local address 127.0.0.1:57180)
0925 10:42:16.012662 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:16.012754 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.61:37529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260925 10:42:16.013842 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:16.013505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.61:37529 (local address 127.0.0.1:57188)
0925 10:42:16.013653 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:16.013729 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.61:37529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 10:42:16.014019 4987 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.61:37529: Network error: Client connection negotiation failed: client connection to 127.23.100.61:37529: connect: Connection refused (error 111)
I20260925 10:42:16.105397 4987 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.61" port: 37529 } attempt: 3
I20260925 10:42:16.106642 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:16.106156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.61:37529 (local address 127.0.0.1:57192)
0925 10:42:16.106403 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:16.106502 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.61:37529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260925 10:42:16.107607 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:16.107270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.61:37529 (local address 127.0.0.1:57196)
0925 10:42:16.107406 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:16.107487 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.61:37529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 10:42:16.107785 4987 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.61:37529: Network error: Client connection negotiation failed: client connection to 127.23.100.61:37529: connect: Connection refused (error 111)
W20260925 10:42:16.229964 4994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:42:16.230296 4994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:42:16.230484 4994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:42:16.237342 4994 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:42:16.237493 4994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:42:16.237599 4994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:42:16.237686 4994 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:42:16.237762 4994 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:42:16.247051 4994 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:33749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.62:37805,127.23.100.61:37529,127.23.100.60:33377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.61.keytab
--principal=kudu/127.23.100.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.61:37529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.100.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:42:16.248243 4994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:42:16.249444 4994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 10:42:16.251000 4987 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.61" port: 37529 } attempt: 4
I20260925 10:42:16.252058 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:16.251567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.61:37529 (local address 127.0.0.1:57210)
0925 10:42:16.251849 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:16.251932 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.61:37529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20260925 10:42:16.252990 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:16.252655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.61:37529 (local address 127.0.0.1:57220)
0925 10:42:16.252785 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:16.252873 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.61:37529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 10:42:16.253157 4987 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.61:37529: Network error: Client connection negotiation failed: client connection to 127.23.100.61:37529: connect: Connection refused (error 111)
W20260925 10:42:16.260932 5000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:42:16.261042 4999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:42:16.261042 5002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:42:16.261543 4994 server_base.cc:1034] running on GCE node
I20260925 10:42:16.262280 4994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:42:16.263677 4994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:42:16.264880 4994 hybrid_clock.cc:648] HybridClock initialized: now 1790332936264829 us; error 34 us; skew 500 ppm
I20260925 10:42:16.265195 4994 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:42:16 dist-test-slave-dxdd krb5kdc[4907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332936, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:42:16.268141 4994 init.cc:377] Logged in from keytab as kudu/127.23.100.61@KRBTEST.COM (short username kudu)
I20260925 10:42:16.269474 4994 webserver.cc:460] Webserver started at http://127.23.100.61:42715/ using document root <none> and password file <none>
I20260925 10:42:16.270097 4994 fs_manager.cc:359] Metadata directory not provided
I20260925 10:42:16.270232 4994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:42:16.270542 4994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:42:16.273376 4994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/minicluster-data/master-1/data/instance:
uuid: "ccbbc4d7911f42e1bcac133c544a12da"
format_stamp: "Formatted at 2026-09-25 10:42:16 on dist-test-slave-dxdd"
server_key: "4ca6e837e72bc3491e75f7f5bf90fecf"
server_key_iv: "78a49d6a83a4c807426b9b53685e1064"
server_key_version: "encryptionkey@0"
I20260925 10:42:16.274106 4994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/minicluster-data/master-1/wal/instance:
uuid: "ccbbc4d7911f42e1bcac133c544a12da"
format_stamp: "Formatted at 2026-09-25 10:42:16 on dist-test-slave-dxdd"
server_key: "4ca6e837e72bc3491e75f7f5bf90fecf"
server_key_iv: "78a49d6a83a4c807426b9b53685e1064"
server_key_version: "encryptionkey@0"
I20260925 10:42:16.278717 4994 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 10:42:16.281759 5009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:42:16.283018 4994 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 10:42:16.283190 4994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/minicluster-data/master-1/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/minicluster-data/master-1/wal
uuid: "ccbbc4d7911f42e1bcac133c544a12da"
format_stamp: "Formatted at 2026-09-25 10:42:16 on dist-test-slave-dxdd"
server_key: "4ca6e837e72bc3491e75f7f5bf90fecf"
server_key_iv: "78a49d6a83a4c807426b9b53685e1064"
server_key_version: "encryptionkey@0"
I20260925 10:42:16.283386 4994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:42:16.307461 4994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:42:16.311272 4994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:42:16.311527 4994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:42:16.328835 4994 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.61:37529
I20260925 10:42:16.328867 5060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.61:37529 every 8 connection(s)
I20260925 10:42:16.330045 4994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/minicluster-data/master-1/data/info.pb
I20260925 10:42:16.333089 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 4994
I20260925 10:42:16.333300 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/minicluster-data/master-1/wal/instance
I20260925 10:42:16.333573 5061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:42:16.333832 23952 external_mini_cluster.cc:1428] Setting key 668cc21dcd01e963345fdddf95bad4e5
I20260925 10:42:16.339699 5061 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.62" port: 37805 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:42:16 dist-test-slave-dxdd krb5kdc[4907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332936, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.61@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.100.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.60@KRBTEST.COM" created.
I20260925 10:42:16.360692 4990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:16.342325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33156 (local address 127.23.100.62:37805)
0925 10:42:16.342524 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:16.342531 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:16.343754 (+ 1223us) server_negotiation.cc:407] Connection header received
0925 10:42:16.345511 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:16.345539 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:16.345664 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:16.345776 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:42:16.347216 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:16.348361 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:16.349787 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:16.350584 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:16.350738 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:16.354154 (+ 3416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:42:16.354159 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:42:16.354162 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:42:16.354195 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:42:16.356936 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:16.357836 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:16.357839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:16.357841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:16.357946 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:16.358394 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:16.358397 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:16.358399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:16.358746 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:42:16.358919 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:16.359974 (+ 1055us) server_negotiation.cc:299] Negotiation successful
0925 10:42:16.360233 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 10:42:16.361212 5064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:16.342794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:37805 (local address 127.0.0.1:33156)
0925 10:42:16.343542 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:16.343595 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 10:42:16.345261 (+ 1666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:42:16.345958 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:42:16.345965 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:42:16.346537 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:42:16.347054 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:16.347070 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:16.348497 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:16.348500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:16.349653 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:16.349664 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:16.350817 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:16.350824 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:16.351125 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:16.351760 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:42:16.351782 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:42:16.353973 (+ 2191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:42:16.357062 (+ 3089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:42:16.357073 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:42:16.357091 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:42:16.357732 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:42:16.358043 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:42:16.358050 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:42:16.358056 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:42:16.358261 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:42:16.359003 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:42:16.359014 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:42:16.359553 (+ 539us) client_negotiation.cc:769] Sending connection context
0925 10:42:16.359832 (+ 279us) client_negotiation.cc:240] Negotiation successful
0925 10:42:16.360179 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":331,"threads_started":1}
I20260925 10:42:16.362850 5061 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.61" port: 37529 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:42:16 dist-test-slave-dxdd krb5kdc[4907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332936, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.61@KRBTEST.COM for kudu/127.23.100.61@KRBTEST.COM
I20260925 10:42:16.377916 5064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:16.363519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.61:37529 (local address 127.0.0.1:57234)
0925 10:42:16.363710 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:16.363734 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 10:42:16.363833 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:42:16.364995 (+ 1162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:42:16.365003 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:42:16.365460 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:42:16.365665 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:16.365679 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:16.367012 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:16.367020 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:16.368617 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:16.368632 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:16.369374 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:16.369385 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:16.369614 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:16.370154 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:42:16.370180 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:42:16.372523 (+ 2343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:42:16.375609 (+ 3086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:42:16.375617 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:42:16.375620 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:42:16.375942 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:42:16.376281 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:42:16.376288 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:42:16.376291 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:42:16.376429 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:42:16.376975 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:42:16.376981 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:42:16.377114 (+ 133us) client_negotiation.cc:769] Sending connection context
0925 10:42:16.377253 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 10:42:16.377515 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 10:42:16.378080 5066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:16.363987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57234 (local address 127.23.100.61:37529)
0925 10:42:16.364618 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:16.364623 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:16.364645 (+ 22us) server_negotiation.cc:407] Connection header received
0925 10:42:16.364683 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:16.364688 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:16.364840 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:16.364949 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:42:16.365812 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:16.366883 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:16.368761 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:16.369232 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:16.369319 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:16.372737 (+ 3418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:42:16.372748 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:42:16.372759 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:42:16.372785 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:42:16.375484 (+ 2699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:16.376070 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:16.376082 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:16.376084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:16.376183 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:16.376528 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:16.376536 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:16.376539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:16.376840 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:42:16.377048 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:16.377395 (+ 347us) server_negotiation.cc:299] Negotiation successful
0925 10:42:16.377673 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":499,"thread_start_us":365,"threads_started":1}
I20260925 10:42:16.380192 5061 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.60" port: 33377 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:42:16.381415 5064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:16.380908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:33377 (local address 127.0.0.1:44286)
0925 10:42:16.381061 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:16.381214 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.60:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 10:42:16.381667 5013 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.100.60:33377: connect: Connection refused (error 111)
I20260925 10:42:16.383092 5064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:16.382807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:33377 (local address 127.0.0.1:44294)
0925 10:42:16.382922 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:16.382995 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.60:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 10:42:16.383299 5061 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.60:33377: Network error: Client connection negotiation failed: client connection to 127.23.100.60:33377: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.60.keytab.
Entry for principal kudu/127.23.100.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.60.keytab.
I20260925 10:42:16.385227 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.60:33377
--webserver_interface=127.23.100.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.60.keytab
--principal=kudu/127.23.100.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:33749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.62:37805,127.23.100.61:37529,127.23.100.60:33377
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:42:16.432626 5061 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.60" port: 33377 } attempt: 1
I20260925 10:42:16.433846 5064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:16.433391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:33377 (local address 127.0.0.1:44296)
0925 10:42:16.433587 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:16.433703 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.60:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260925 10:42:16.434902 5064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:16.434619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:33377 (local address 127.0.0.1:44298)
0925 10:42:16.434739 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:16.434812 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.60:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 10:42:16.435153 5061 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.60:33377: Network error: Client connection negotiation failed: client connection to 127.23.100.60:33377: connect: Connection refused (error 111)
I20260925 10:42:16.503340 5061 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.60" port: 33377 } attempt: 2
I20260925 10:42:16.504442 5064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:16.504026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:33377 (local address 127.0.0.1:44306)
0925 10:42:16.504202 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:16.504292 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.60:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 10:42:16.505323 5064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:16.505027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:33377 (local address 127.0.0.1:44312)
0925 10:42:16.505159 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:16.505226 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.60:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 10:42:16.505512 5061 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.60:33377: Network error: Client connection negotiation failed: client connection to 127.23.100.60:33377: connect: Connection refused (error 111)
I20260925 10:42:16.552424 4987 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.61" port: 37529 } attempt: 5
Sep 25 10:42:16 dist-test-slave-dxdd krb5kdc[4907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332935, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.62@KRBTEST.COM for kudu/127.23.100.61@KRBTEST.COM
I20260925 10:42:16.566730 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:16.553149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.61:37529 (local address 127.0.0.1:57246)
0925 10:42:16.553340 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:16.553376 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 10:42:16.553505 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:42:16.553865 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:42:16.553869 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:42:16.554246 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:42:16.554491 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:16.554501 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:16.555917 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:16.555921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:16.556744 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:16.556754 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:16.557861 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:16.557866 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:16.557995 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:16.558683 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:42:16.558700 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:42:16.560921 (+ 2221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:42:16.564522 (+ 3601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:42:16.564526 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:42:16.564528 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:42:16.564873 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:42:16.565270 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:42:16.565273 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:42:16.565275 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:42:16.565387 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:42:16.566036 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:42:16.566039 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:42:16.566160 (+ 121us) client_negotiation.cc:769] Sending connection context
0925 10:42:16.566263 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 10:42:16.566493 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 10:42:16.567101 5066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:16.553241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57246 (local address 127.23.100.61:37529)
0925 10:42:16.553447 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:16.553453 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:16.553494 (+ 41us) server_negotiation.cc:407] Connection header received
0925 10:42:16.553632 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:16.553637 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:16.553715 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:16.553834 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:42:16.554677 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:16.555802 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:16.556896 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:16.557720 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:16.557872 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:16.561110 (+ 3238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:42:16.561117 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:42:16.561121 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:42:16.561159 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:42:16.564398 (+ 3239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:16.564990 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:16.564998 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:16.565003 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:16.565163 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:16.565489 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:16.565497 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:16.565501 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:16.565924 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:42:16.566093 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:16.566438 (+ 345us) server_negotiation.cc:299] Negotiation successful
0925 10:42:16.566707 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 10:42:16.568092 4987 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.60" port: 33377 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:42:16.569298 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:16.568913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:33377 (local address 127.0.0.1:44314)
0925 10:42:16.569071 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:16.569153 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.60:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 10:42:16.570235 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:16.569928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:33377 (local address 127.0.0.1:44322)
0925 10:42:16.570062 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:16.570130 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.60:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 10:42:16.570436 4987 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.60:33377: Network error: Client connection negotiation failed: client connection to 127.23.100.60:33377: connect: Connection refused (error 111)
I20260925 10:42:16.596779 5061 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.60" port: 33377 } attempt: 3
I20260925 10:42:16.597812 5064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:16.597361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:33377 (local address 127.0.0.1:44338)
0925 10:42:16.597576 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:16.597665 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.60:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 10:42:16.598795 5064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:16.598408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:33377 (local address 127.0.0.1:44354)
0925 10:42:16.598619 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:16.598686 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.60:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20260925 10:42:16.598986 5061 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.60:33377: Network error: Client connection negotiation failed: client connection to 127.23.100.60:33377: connect: Connection refused (error 111)
I20260925 10:42:16.621645 4987 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.60" port: 33377 } attempt: 1
I20260925 10:42:16.622659 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:16.622217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:33377 (local address 127.0.0.1:44358)
0925 10:42:16.622420 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:16.622509 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.60:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 10:42:16.623620 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:16.623318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:33377 (local address 127.0.0.1:44368)
0925 10:42:16.623471 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:16.623533 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.60:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260925 10:42:16.623811 4987 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.60:33377: Network error: Client connection negotiation failed: client connection to 127.23.100.60:33377: connect: Connection refused (error 111)
I20260925 10:42:16.691994 4987 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.60" port: 33377 } attempt: 2
I20260925 10:42:16.693060 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:16.692625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:33377 (local address 127.0.0.1:44372)
0925 10:42:16.692807 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:16.692911 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.60:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 10:42:16.693989 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:16.693690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:33377 (local address 127.0.0.1:44386)
0925 10:42:16.693823 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:16.693899 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.60:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 10:42:16.694195 4987 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.60:33377: Network error: Client connection negotiation failed: client connection to 127.23.100.60:33377: connect: Connection refused (error 111)
W20260925 10:42:16.703583 5068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:42:16.703872 5068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:42:16.704035 5068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:42:16.710860 5068 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:42:16.710980 5068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:42:16.711073 5068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:42:16.711167 5068 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:42:16.711238 5068 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:42:16.720961 5068 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:33749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.62:37805,127.23.100.61:37529,127.23.100.60:33377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.60.keytab
--principal=kudu/127.23.100.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.60:33377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.100.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:42:16.722272 5068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:42:16.723555 5068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:42:16.735072 5076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:42:16.735169 5074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:42:16.735172 5073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:42:16.739619 5068 server_base.cc:1034] running on GCE node
I20260925 10:42:16.740351 5068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:42:16.741698 5068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:42:16.742169 5061 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.60" port: 33377 } attempt: 4
I20260925 10:42:16.742918 5068 hybrid_clock.cc:648] HybridClock initialized: now 1790332936742873 us; error 41 us; skew 500 ppm
I20260925 10:42:16.743229 5064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:16.742742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:33377 (local address 127.0.0.1:44394)
0925 10:42:16.742959 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:16.743059 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.60:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 10:42:16.743242 5068 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:42:16.744112 5064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:16.743805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:33377 (local address 127.0.0.1:44406)
0925 10:42:16.743947 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:16.744018 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.60:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 10:42:16.744374 5061 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.60:33377: Network error: Client connection negotiation failed: client connection to 127.23.100.60:33377: connect: Connection refused (error 111)
Sep 25 10:42:16 dist-test-slave-dxdd krb5kdc[4907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332936, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:42:16.746282 5068 init.cc:377] Logged in from keytab as kudu/127.23.100.60@KRBTEST.COM (short username kudu)
I20260925 10:42:16.747668 5068 webserver.cc:460] Webserver started at http://127.23.100.60:44003/ using document root <none> and password file <none>
I20260925 10:42:16.748306 5068 fs_manager.cc:359] Metadata directory not provided
I20260925 10:42:16.748402 5068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:42:16.748644 5068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:42:16.751446 5068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/minicluster-data/master-2/data/instance:
uuid: "d3dc9bd021b146eba95ae0ce6e015042"
format_stamp: "Formatted at 2026-09-25 10:42:16 on dist-test-slave-dxdd"
server_key: "2e16f66a2f7522cf4a5f9a5f5c9b713d"
server_key_iv: "645d62c3d639104f2cf75ebf41891024"
server_key_version: "encryptionkey@0"
I20260925 10:42:16.752246 5068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/minicluster-data/master-2/wal/instance:
uuid: "d3dc9bd021b146eba95ae0ce6e015042"
format_stamp: "Formatted at 2026-09-25 10:42:16 on dist-test-slave-dxdd"
server_key: "2e16f66a2f7522cf4a5f9a5f5c9b713d"
server_key_iv: "645d62c3d639104f2cf75ebf41891024"
server_key_version: "encryptionkey@0"
I20260925 10:42:16.757107 5068 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 10:42:16.760211 5083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:42:16.761458 5068 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 10:42:16.761590 5068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/minicluster-data/master-2/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/minicluster-data/master-2/wal
uuid: "d3dc9bd021b146eba95ae0ce6e015042"
format_stamp: "Formatted at 2026-09-25 10:42:16 on dist-test-slave-dxdd"
server_key: "2e16f66a2f7522cf4a5f9a5f5c9b713d"
server_key_iv: "645d62c3d639104f2cf75ebf41891024"
server_key_version: "encryptionkey@0"
I20260925 10:42:16.761750 5068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:42:16.775337 5068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:42:16.779089 5068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:42:16.779307 5068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:42:16.798712 5068 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.60:33377
I20260925 10:42:16.798734 5134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.60:33377 every 8 connection(s)
I20260925 10:42:16.800091 5068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/minicluster-data/master-2/data/info.pb
I20260925 10:42:16.800436 4987 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.60" port: 33377 } attempt: 3
I20260925 10:42:16.805236 5135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:42:16.805893 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 5068
I20260925 10:42:16.806102 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790332740062128-23952-0/minicluster-data/master-2/wal/instance
I20260925 10:42:16.806664 23952 external_mini_cluster.cc:1428] Setting key 043cdc40055f08e56075b07576b15b17
Sep 25 10:42:16 dist-test-slave-dxdd krb5kdc[4907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332935, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.62@KRBTEST.COM for kudu/127.23.100.60@KRBTEST.COM
I20260925 10:42:16.817243 5135 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.62" port: 37805 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:42:16.828119 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:16.801192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:33377 (local address 127.0.0.1:44408)
0925 10:42:16.801459 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:16.801479 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:42:16.801616 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:42:16.806018 (+ 4402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:42:16.806027 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:42:16.806551 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:42:16.806766 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:16.806779 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:16.809662 (+ 2883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:16.809671 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:16.811660 (+ 1989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:16.811681 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:16.813589 (+ 1908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:16.813598 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:16.813851 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:16.814708 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:42:16.814741 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:42:16.818533 (+ 3792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:42:16.824988 (+ 6455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:42:16.824993 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:42:16.824997 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:42:16.825497 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:42:16.825962 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:42:16.825968 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:42:16.825972 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:42:16.826137 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:42:16.827095 (+ 958us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:42:16.827101 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:42:16.827268 (+ 167us) client_negotiation.cc:769] Sending connection context
0925 10:42:16.827440 (+ 172us) client_negotiation.cc:240] Negotiation successful
0925 10:42:16.827695 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260925 10:42:16.829844 5137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:16.802209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44408 (local address 127.23.100.60:33377)
0925 10:42:16.803189 (+ 980us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:16.803200 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:16.803239 (+ 39us) server_negotiation.cc:407] Connection header received
0925 10:42:16.804950 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:16.804970 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:16.805558 (+ 588us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:16.806030 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:42:16.806976 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:16.809436 (+ 2460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:16.811906 (+ 2470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:16.813351 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:16.818821 (+ 5470us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:16.819342 (+ 521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:42:16.819355 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:42:16.819378 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:42:16.819465 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:42:16.824837 (+ 5372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:16.825644 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:16.825652 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:16.825660 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:16.825836 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:16.826270 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:16.826275 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:16.826278 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:16.826950 (+ 672us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:42:16.827799 (+ 849us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:16.828102 (+ 303us) server_negotiation.cc:299] Negotiation successful
0925 10:42:16.828497 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":402,"threads_started":1}
Sep 25 10:42:16 dist-test-slave-dxdd krb5kdc[4907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332936, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.60@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
Sep 25 10:42:16 dist-test-slave-dxdd krb5kdc[4907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.60@KRBTEST.COM
I20260925 10:42:16.837855 4987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 355e9e0b4d5e409596876e18547a020b: Bootstrap starting.
I20260925 10:42:16.839064 4990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:16.821946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33166 (local address 127.23.100.62:37805)
0925 10:42:16.822120 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:16.822126 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:16.822663 (+ 537us) server_negotiation.cc:407] Connection header received
0925 10:42:16.823165 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:16.823173 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:16.823250 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:16.823355 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:42:16.824875 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:16.825954 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:16.828001 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:16.828606 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:16.828723 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:16.832215 (+ 3492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:42:16.832222 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:42:16.832228 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:42:16.832268 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:42:16.835875 (+ 3607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:16.836790 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:16.836795 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:16.836799 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:16.836974 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:16.837378 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:16.837387 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:16.837390 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:16.837715 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:42:16.837851 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:16.838522 (+ 671us) server_negotiation.cc:299] Negotiation successful
0925 10:42:16.838716 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 10:42:16.840190 5140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:16.821494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:37805 (local address 127.0.0.1:33166)
0925 10:42:16.822512 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:16.822573 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 10:42:16.822744 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:42:16.823429 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:42:16.823437 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:42:16.824354 (+ 917us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:42:16.824735 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:16.824745 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:16.826110 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:16.826115 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:16.827839 (+ 1724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:16.827852 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:16.829031 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:16.829036 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:16.829148 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:16.829691 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:42:16.829707 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:42:16.831961 (+ 2254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:42:16.836047 (+ 4086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:42:16.836058 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:42:16.836065 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:42:16.836652 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:42:16.837112 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:42:16.837116 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:42:16.837120 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:42:16.837268 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:42:16.838004 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:42:16.838013 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:42:16.838180 (+ 167us) client_negotiation.cc:769] Sending connection context
0925 10:42:16.839507 (+ 1327us) client_negotiation.cc:240] Negotiation successful
0925 10:42:16.839792 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":748,"mutex_wait_us":35,"thread_start_us":441,"threads_started":1}
I20260925 10:42:16.841737 5138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:16.807676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44414 (local address 127.23.100.60:33377)
0925 10:42:16.821980 (+ 14304us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:16.821990 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:16.822016 (+ 26us) server_negotiation.cc:407] Connection header received
0925 10:42:16.822070 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:16.822075 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:16.822146 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:16.822277 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:42:16.824314 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:16.825485 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:16.827076 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:16.827573 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:16.827742 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:16.834278 (+ 6536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:42:16.834286 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:42:16.834290 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:42:16.834352 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:42:16.837843 (+ 3491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:16.838774 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:16.838779 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:16.838782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:16.838925 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:16.839370 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:16.839375 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:16.839378 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:16.839771 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:42:16.840074 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:16.841156 (+ 1082us) server_negotiation.cc:299] Negotiation successful
0925 10:42:16.841364 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":14122,"thread_start_us":13937,"threads_started":1}
I20260925 10:42:16.841922 4987 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 355e9e0b4d5e409596876e18547a020b: Neither blocks nor log segments found. Creating new log.
I20260925 10:42:16.843173 4987 log.cc:824] T 00000000000000000000000000000000 P 355e9e0b4d5e409596876e18547a020b: Log is configured to *not* fsync() on all Append() calls
I20260925 10:42:16.848327 5135 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.61" port: 37529 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:42:16.850157 4987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 355e9e0b4d5e409596876e18547a020b: No bootstrap required, opened a new log
Sep 25 10:42:16 dist-test-slave-dxdd krb5kdc[4907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332936, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.60@KRBTEST.COM for kudu/127.23.100.61@KRBTEST.COM
I20260925 10:42:16.862884 4987 raft_consensus.cc:348] T 00000000000000000000000000000000 P 355e9e0b4d5e409596876e18547a020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355e9e0b4d5e409596876e18547a020b" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 37805 } } peers { permanent_uuid: "ccbbc4d7911f42e1bcac133c544a12da" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 37529 } } peers { permanent_uuid: "d3dc9bd021b146eba95ae0ce6e015042" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 33377 } }
I20260925 10:42:16.863282 4987 raft_consensus.cc:374] T 00000000000000000000000000000000 P 355e9e0b4d5e409596876e18547a020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:42:16.863395 4987 raft_consensus.cc:729] T 00000000000000000000000000000000 P 355e9e0b4d5e409596876e18547a020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 355e9e0b4d5e409596876e18547a020b, State: Initialized, Role: FOLLOWER
I20260925 10:42:16.864032 4987 consensus_queue.cc:260] T 00000000000000000000000000000000 P 355e9e0b4d5e409596876e18547a020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355e9e0b4d5e409596876e18547a020b" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 37805 } } peers { permanent_uuid: "ccbbc4d7911f42e1bcac133c544a12da" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 37529 } } peers { permanent_uuid: "d3dc9bd021b146eba95ae0ce6e015042" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 33377 } }
Sep 25 10:42:16 dist-test-slave-dxdd krb5kdc[4907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:42:16.865180 5142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 355e9e0b4d5e409596876e18547a020b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355e9e0b4d5e409596876e18547a020b" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 37805 } } peers { permanent_uuid: "ccbbc4d7911f42e1bcac133c544a12da" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 37529 } } peers { permanent_uuid: "d3dc9bd021b146eba95ae0ce6e015042" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 33377 } } }
I20260925 10:42:16.865795 5142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 355e9e0b4d5e409596876e18547a020b [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:42:16.866631 4987 sys_catalog.cc:564] T 00000000000000000000000000000000 P 355e9e0b4d5e409596876e18547a020b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:42:16.867691 5066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:16.849095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57258 (local address 127.23.100.61:37529)
0925 10:42:16.849269 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:16.849275 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:16.852671 (+ 3396us) server_negotiation.cc:407] Connection header received
0925 10:42:16.852844 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:16.852849 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:16.852935 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:16.853040 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:42:16.853974 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:16.854783 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:16.856505 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:16.857116 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:16.857210 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:16.861090 (+ 3880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:42:16.861097 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:42:16.861102 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:42:16.861137 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:42:16.864140 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:16.865028 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:16.865034 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:16.865037 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:16.865237 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:16.865849 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:16.865857 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:16.865862 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:16.866205 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:42:16.866375 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:16.867120 (+ 745us) server_negotiation.cc:299] Negotiation successful
0925 10:42:16.867333 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 10:42:16.867908 5140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:16.848991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.61:37529 (local address 127.0.0.1:57258)
0925 10:42:16.852561 (+ 3570us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:16.852576 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:42:16.852710 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:42:16.853230 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:42:16.853235 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:42:16.853683 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:42:16.853827 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:16.853835 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:16.854971 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:16.854977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:16.856334 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:16.856347 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:16.857462 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:16.857468 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:16.857680 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:16.858299 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:42:16.858322 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:42:16.860898 (+ 2576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:42:16.864289 (+ 3391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:42:16.864294 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:42:16.864297 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:42:16.864830 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:42:16.865486 (+ 656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:42:16.865491 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:42:16.865494 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:42:16.865709 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:42:16.866702 (+ 993us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:42:16.866706 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:42:16.866864 (+ 158us) client_negotiation.cc:769] Sending connection context
0925 10:42:16.867007 (+ 143us) client_negotiation.cc:240] Negotiation successful
0925 10:42:16.867259 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3483}
I20260925 10:42:16.869495 5135 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.60" port: 33377 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:42:16.872915 4990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:16.853114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33174 (local address 127.23.100.62:37805)
0925 10:42:16.853275 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:16.853286 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:16.853306 (+ 20us) server_negotiation.cc:407] Connection header received
0925 10:42:16.853353 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:16.853362 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:16.853456 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:16.853562 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:42:16.855087 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:16.856213 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:16.858774 (+ 2561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:16.859638 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:16.859735 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:16.865760 (+ 6025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:42:16.865768 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:42:16.865771 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:42:16.865814 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:42:16.869279 (+ 3465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:16.870258 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:16.870265 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:16.870270 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:16.870460 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:16.870922 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:16.870925 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:16.870927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:16.871169 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:42:16.871469 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:16.872345 (+ 876us) server_negotiation.cc:299] Negotiation successful
0925 10:42:16.872554 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Sep 25 10:42:16 dist-test-slave-dxdd krb5kdc[4907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332936, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.60@KRBTEST.COM for kudu/127.23.100.60@KRBTEST.COM
Sep 25 10:42:16 dist-test-slave-dxdd krb5kdc[4907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.61@KRBTEST.COM
I20260925 10:42:16.887064 5140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:16.871259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:33377 (local address 127.0.0.1:44430)
0925 10:42:16.871631 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:16.871647 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:42:16.871738 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:42:16.872485 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:42:16.872490 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:42:16.872908 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:42:16.873128 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:16.873139 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:16.874191 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:16.874197 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:16.875444 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:16.875458 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:16.876726 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:16.876731 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:16.876956 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:16.877594 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:42:16.877616 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:42:16.880078 (+ 2462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:42:16.884102 (+ 4024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:42:16.884106 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:42:16.884110 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:42:16.884601 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:42:16.884983 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:42:16.884987 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:42:16.884990 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:42:16.885115 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:42:16.885801 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:42:16.885806 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:42:16.885972 (+ 166us) client_negotiation.cc:769] Sending connection context
0925 10:42:16.886127 (+ 155us) client_negotiation.cc:240] Negotiation successful
0925 10:42:16.886674 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234}
I20260925 10:42:16.887766 5138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:16.871931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44430 (local address 127.23.100.60:33377)
0925 10:42:16.872150 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:16.872159 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:16.872180 (+ 21us) server_negotiation.cc:407] Connection header received
0925 10:42:16.872234 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:16.872241 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:16.872323 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:16.872450 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:42:16.873248 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:16.874028 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:16.875594 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:16.876518 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:16.876673 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:16.880224 (+ 3551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:42:16.880230 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:42:16.880235 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:42:16.880267 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:42:16.883958 (+ 3691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:16.884721 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:16.884728 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:16.884732 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:16.884849 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:16.885230 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:16.885237 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:16.885240 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:16.885610 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:42:16.885753 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:16.886461 (+ 708us) server_negotiation.cc:299] Negotiation successful
0925 10:42:16.887438 (+ 977us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":757,"server-negotiator.queue_time_us":60}
I20260925 10:42:16.892946 5066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:16.875087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57262 (local address 127.23.100.61:37529)
0925 10:42:16.875222 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:16.875227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:16.875443 (+ 216us) server_negotiation.cc:407] Connection header received
0925 10:42:16.875657 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:16.875661 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:16.875720 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:16.875816 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:42:16.877097 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:16.878082 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:16.879979 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:16.880658 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:16.880758 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:16.886287 (+ 5529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:42:16.886293 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:42:16.886297 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:42:16.886348 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:42:16.889860 (+ 3512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:16.890706 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:16.890711 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:16.890714 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:16.890867 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:16.891306 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:16.891311 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:16.891314 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:16.891616 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:42:16.891825 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:16.892418 (+ 593us) server_negotiation.cc:299] Negotiation successful
0925 10:42:16.892642 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 10:42:16.893509 5135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3dc9bd021b146eba95ae0ce6e015042: Bootstrap starting.
I20260925 10:42:16.897979 5135 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3dc9bd021b146eba95ae0ce6e015042: Neither blocks nor log segments found. Creating new log.
I20260925 10:42:16.898554 23952 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:42:16.899205 5135 log.cc:824] T 00000000000000000000000000000000 P d3dc9bd021b146eba95ae0ce6e015042: Log is configured to *not* fsync() on all Append() calls
I20260925 10:42:16.883605 4920 master_runner.cc:186] Error getting master registration for 127.23.100.61:37529: OK, instance_id { permanent_uuid: "ccbbc4d7911f42e1bcac133c544a12da" instance_seqno: 1790332936323880 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 10:42:16.899627 4920 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 10:42:16.902251 5153 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 355e9e0b4d5e409596876e18547a020b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:42:16.902377 5153 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:42:16.908742 5135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3dc9bd021b146eba95ae0ce6e015042: No bootstrap required, opened a new log
I20260925 10:42:16.920133 5135 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3dc9bd021b146eba95ae0ce6e015042 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355e9e0b4d5e409596876e18547a020b" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 37805 } } peers { permanent_uuid: "ccbbc4d7911f42e1bcac133c544a12da" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 37529 } } peers { permanent_uuid: "d3dc9bd021b146eba95ae0ce6e015042" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 33377 } }
I20260925 10:42:16.920567 5135 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3dc9bd021b146eba95ae0ce6e015042 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:42:16.920706 5135 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3dc9bd021b146eba95ae0ce6e015042 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3dc9bd021b146eba95ae0ce6e015042, State: Initialized, Role: FOLLOWER
I20260925 10:42:16.921548 5135 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3dc9bd021b146eba95ae0ce6e015042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355e9e0b4d5e409596876e18547a020b" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 37805 } } peers { permanent_uuid: "ccbbc4d7911f42e1bcac133c544a12da" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 37529 } } peers { permanent_uuid: "d3dc9bd021b146eba95ae0ce6e015042" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 33377 } }
I20260925 10:42:16.933426 5163 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3dc9bd021b146eba95ae0ce6e015042 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355e9e0b4d5e409596876e18547a020b" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 37805 } } peers { permanent_uuid: "ccbbc4d7911f42e1bcac133c544a12da" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 37529 } } peers { permanent_uuid: "d3dc9bd021b146eba95ae0ce6e015042" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 33377 } } }
I20260925 10:42:16.933842 5163 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3dc9bd021b146eba95ae0ce6e015042 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:42:16.934132 4990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:16.910531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33190 (local address 127.23.100.62:37805)
0925 10:42:16.910686 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:16.910694 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:16.910714 (+ 20us) server_negotiation.cc:407] Connection header received
0925 10:42:16.910775 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:16.910780 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:16.910866 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:16.910987 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:42:16.913451 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:16.914577 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:16.917456 (+ 2879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:16.918141 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:16.918248 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:16.925509 (+ 7261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:42:16.925517 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:42:16.925524 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:42:16.925574 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:42:16.929870 (+ 4296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:16.930738 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:16.930743 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:16.930746 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:16.930920 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:16.931695 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:16.931701 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:16.931705 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:16.932052 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:42:16.932328 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:16.933326 (+ 998us) server_negotiation.cc:299] Negotiation successful
0925 10:42:16.933595 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 10:42:16.935781 5135 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3dc9bd021b146eba95ae0ce6e015042 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:42:16.937849 5066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:16.908853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57266 (local address 127.23.100.61:37529)
0925 10:42:16.909004 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:16.909009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:16.920065 (+ 11056us) server_negotiation.cc:407] Connection header received
0925 10:42:16.920541 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:16.920547 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:16.920636 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:16.920701 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:42:16.922457 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:16.923561 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:16.925521 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:16.926135 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:16.926228 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:16.930760 (+ 4532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:42:16.930766 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:42:16.930771 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:42:16.930809 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:42:16.934269 (+ 3460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:16.935411 (+ 1142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:16.935416 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:16.935420 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:16.935580 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:16.936056 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:16.936062 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:16.936065 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:16.936445 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:42:16.936692 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:16.937260 (+ 568us) server_negotiation.cc:299] Negotiation successful
0925 10:42:16.937498 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 10:42:16.946794 5138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:16.910883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44436 (local address 127.23.100.60:33377)
0925 10:42:16.911238 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:16.911243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:16.925047 (+ 13804us) server_negotiation.cc:407] Connection header received
0925 10:42:16.925245 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:16.925249 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:16.925318 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:16.925406 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:42:16.929469 (+ 4063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:16.930521 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:16.937107 (+ 6586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:16.937790 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:16.937910 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:16.941209 (+ 3299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:42:16.941214 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:42:16.941217 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:42:16.941247 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:42:16.944007 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:16.944730 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:16.944734 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:16.944736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:16.944867 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:16.945279 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:16.945283 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:16.945285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:16.945580 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:42:16.945819 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:16.946164 (+ 345us) server_negotiation.cc:299] Negotiation successful
0925 10:42:16.946431 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247}
W20260925 10:42:16.966827 5174 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d3dc9bd021b146eba95ae0ce6e015042: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:42:16.967029 5174 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:42:16.960739 5068 master_runner.cc:186] Error getting master registration for 127.23.100.61:37529: OK, instance_id { permanent_uuid: "ccbbc4d7911f42e1bcac133c544a12da" instance_seqno: 1790332936323880 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 10:42:16.971956 5068 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 10:42:17.031227 5089 tablet.cc:2369] T 00000000000000000000000000000000 P d3dc9bd021b146eba95ae0ce6e015042: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:42:17.043605 5061 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.60" port: 33377 } attempt: 5
Sep 25 10:42:17 dist-test-slave-dxdd krb5kdc[4907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332936, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.61@KRBTEST.COM for kudu/127.23.100.60@KRBTEST.COM
I20260925 10:42:17.063681 5138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:17.044265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44438 (local address 127.23.100.60:33377)
0925 10:42:17.044442 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:17.044448 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:17.044544 (+ 96us) server_negotiation.cc:407] Connection header received
0925 10:42:17.044700 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:17.044705 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:17.044778 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:17.044882 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:42:17.045883 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:17.047103 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:17.048878 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:17.049708 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:17.049827 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:17.054436 (+ 4609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:42:17.054443 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:42:17.054448 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:42:17.054491 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:42:17.060696 (+ 6205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:17.061522 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:17.061528 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:17.061532 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:17.061680 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:17.062125 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:17.062131 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:17.062134 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:17.062525 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:42:17.062692 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:17.063098 (+ 406us) server_negotiation.cc:299] Negotiation successful
0925 10:42:17.063322 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 10:42:17.063681 5064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:17.044186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:33377 (local address 127.0.0.1:44438)
0925 10:42:17.044451 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:17.044470 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:42:17.044585 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:42:17.044929 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:42:17.044934 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:42:17.045503 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:42:17.045712 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:17.045724 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:17.047269 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:17.047274 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:17.048709 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:17.048723 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:17.050033 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:17.050040 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:17.050282 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:17.051067 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:42:17.051093 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:42:17.054210 (+ 3117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:42:17.060838 (+ 6628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:42:17.060844 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:42:17.060848 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:42:17.061379 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:42:17.061804 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:42:17.061809 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:42:17.061813 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:42:17.062005 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:42:17.062649 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:42:17.062654 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:42:17.062808 (+ 154us) client_negotiation.cc:769] Sending connection context
0925 10:42:17.062987 (+ 179us) client_negotiation.cc:240] Negotiation successful
0925 10:42:17.063275 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132}
I20260925 10:42:17.070173 5061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccbbc4d7911f42e1bcac133c544a12da: Bootstrap starting.
I20260925 10:42:17.076272 5061 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccbbc4d7911f42e1bcac133c544a12da: Neither blocks nor log segments found. Creating new log.
I20260925 10:42:17.077699 5061 log.cc:824] T 00000000000000000000000000000000 P ccbbc4d7911f42e1bcac133c544a12da: Log is configured to *not* fsync() on all Append() calls
I20260925 10:42:17.080940 5061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccbbc4d7911f42e1bcac133c544a12da: No bootstrap required, opened a new log
I20260925 10:42:17.091794 5061 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccbbc4d7911f42e1bcac133c544a12da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355e9e0b4d5e409596876e18547a020b" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 37805 } } peers { permanent_uuid: "ccbbc4d7911f42e1bcac133c544a12da" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 37529 } } peers { permanent_uuid: "d3dc9bd021b146eba95ae0ce6e015042" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 33377 } }
I20260925 10:42:17.092188 5061 raft_consensus.cc:374] T 00000000000000000000000000000000 P ccbbc4d7911f42e1bcac133c544a12da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:42:17.092300 5061 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccbbc4d7911f42e1bcac133c544a12da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccbbc4d7911f42e1bcac133c544a12da, State: Initialized, Role: FOLLOWER
W20260925 10:42:17.092761 4941 tablet.cc:2369] T 00000000000000000000000000000000 P 355e9e0b4d5e409596876e18547a020b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:42:17.093055 5061 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccbbc4d7911f42e1bcac133c544a12da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355e9e0b4d5e409596876e18547a020b" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 37805 } } peers { permanent_uuid: "ccbbc4d7911f42e1bcac133c544a12da" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 37529 } } peers { permanent_uuid: "d3dc9bd021b146eba95ae0ce6e015042" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 33377 } }
I20260925 10:42:17.097862 5177 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccbbc4d7911f42e1bcac133c544a12da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355e9e0b4d5e409596876e18547a020b" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 37805 } } peers { permanent_uuid: "ccbbc4d7911f42e1bcac133c544a12da" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 37529 } } peers { permanent_uuid: "d3dc9bd021b146eba95ae0ce6e015042" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 33377 } } }
I20260925 10:42:17.098333 5177 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccbbc4d7911f42e1bcac133c544a12da [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:42:17.099181 5061 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccbbc4d7911f42e1bcac133c544a12da [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:42:17.112680 4994 master_runner.cc:263] No leader master found from master ccbbc4d7911f42e1bcac133c544a12da
I20260925 10:42:17.112823 4994 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 10:42:17.129163 5188 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ccbbc4d7911f42e1bcac133c544a12da: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:42:17.129318 5188 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:42:17.174594 5142 raft_consensus.cc:482] T 00000000000000000000000000000000 P 355e9e0b4d5e409596876e18547a020b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:42:17.174959 5142 raft_consensus.cc:504] T 00000000000000000000000000000000 P 355e9e0b4d5e409596876e18547a020b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355e9e0b4d5e409596876e18547a020b" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 37805 } } peers { permanent_uuid: "ccbbc4d7911f42e1bcac133c544a12da" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 37529 } } peers { permanent_uuid: "d3dc9bd021b146eba95ae0ce6e015042" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 33377 } }
I20260925 10:42:17.176672 5142 leader_election.cc:288] T 00000000000000000000000000000000 P 355e9e0b4d5e409596876e18547a020b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ccbbc4d7911f42e1bcac133c544a12da (127.23.100.61:37529), d3dc9bd021b146eba95ae0ce6e015042 (127.23.100.60:33377)
I20260925 10:42:17.177913 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "355e9e0b4d5e409596876e18547a020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccbbc4d7911f42e1bcac133c544a12da" is_pre_election: true
I20260925 10:42:17.178685 5036 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ccbbc4d7911f42e1bcac133c544a12da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 355e9e0b4d5e409596876e18547a020b in term 0.
I20260925 10:42:17.179832 5110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "355e9e0b4d5e409596876e18547a020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3dc9bd021b146eba95ae0ce6e015042" is_pre_election: true
I20260925 10:42:17.180402 5110 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d3dc9bd021b146eba95ae0ce6e015042 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 355e9e0b4d5e409596876e18547a020b in term 0.
I20260925 10:42:17.181210 4937 leader_election.cc:302] T 00000000000000000000000000000000 P 355e9e0b4d5e409596876e18547a020b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 355e9e0b4d5e409596876e18547a020b, ccbbc4d7911f42e1bcac133c544a12da; no voters:
I20260925 10:42:17.182080 5142 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 355e9e0b4d5e409596876e18547a020b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:42:17.182250 5142 raft_consensus.cc:482] T 00000000000000000000000000000000 P 355e9e0b4d5e409596876e18547a020b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:42:17.182456 5142 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 355e9e0b4d5e409596876e18547a020b [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:42:17.185818 5142 raft_consensus.cc:504] T 00000000000000000000000000000000 P 355e9e0b4d5e409596876e18547a020b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355e9e0b4d5e409596876e18547a020b" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 37805 } } peers { permanent_uuid: "ccbbc4d7911f42e1bcac133c544a12da" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 37529 } } peers { permanent_uuid: "d3dc9bd021b146eba95ae0ce6e015042" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 33377 } }
I20260925 10:42:17.192735 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "355e9e0b4d5e409596876e18547a020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccbbc4d7911f42e1bcac133c544a12da"
I20260925 10:42:17.192976 5036 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccbbc4d7911f42e1bcac133c544a12da [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:42:17.196554 5036 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ccbbc4d7911f42e1bcac133c544a12da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 355e9e0b4d5e409596876e18547a020b in term 1.
I20260925 10:42:17.197081 4937 leader_election.cc:302] T 00000000000000000000000000000000 P 355e9e0b4d5e409596876e18547a020b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 355e9e0b4d5e409596876e18547a020b, ccbbc4d7911f42e1bcac133c544a12da; no voters:
I20260925 10:42:17.200253 5110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "355e9e0b4d5e409596876e18547a020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3dc9bd021b146eba95ae0ce6e015042"
I20260925 10:42:17.200485 5110 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3dc9bd021b146eba95ae0ce6e015042 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:42:17.204049 5110 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d3dc9bd021b146eba95ae0ce6e015042 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 355e9e0b4d5e409596876e18547a020b in term 1.
I20260925 10:42:17.204764 5142 leader_election.cc:288] T 00000000000000000000000000000000 P 355e9e0b4d5e409596876e18547a020b [CANDIDATE]: Term 1 election: Requested vote from peers ccbbc4d7911f42e1bcac133c544a12da (127.23.100.61:37529), d3dc9bd021b146eba95ae0ce6e015042 (127.23.100.60:33377)
I20260925 10:42:17.205154 5142 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 355e9e0b4d5e409596876e18547a020b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:42:17.205588 5142 raft_consensus.cc:686] T 00000000000000000000000000000000 P 355e9e0b4d5e409596876e18547a020b [term 1 LEADER]: Becoming Leader. State: Replica: 355e9e0b4d5e409596876e18547a020b, State: Running, Role: LEADER
I20260925 10:42:17.206881 5142 consensus_queue.cc:237] T 00000000000000000000000000000000 P 355e9e0b4d5e409596876e18547a020b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355e9e0b4d5e409596876e18547a020b" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 37805 } } peers { permanent_uuid: "ccbbc4d7911f42e1bcac133c544a12da" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 37529 } } peers { permanent_uuid: "d3dc9bd021b146eba95ae0ce6e015042" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 33377 } }
I20260925 10:42:17.224874 5190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 355e9e0b4d5e409596876e18547a020b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 355e9e0b4d5e409596876e18547a020b. Latest consensus state: current_term: 1 leader_uuid: "355e9e0b4d5e409596876e18547a020b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355e9e0b4d5e409596876e18547a020b" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 37805 } } peers { permanent_uuid: "ccbbc4d7911f42e1bcac133c544a12da" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 37529 } } peers { permanent_uuid: "d3dc9bd021b146eba95ae0ce6e015042" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 33377 } } }
I20260925 10:42:17.225481 5190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 355e9e0b4d5e409596876e18547a020b [sys.catalog]: This master's current role is: LEADER
I20260925 10:42:17.234055 5192 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:42:17.236582 5192 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:42:17.257726 5110 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d3dc9bd021b146eba95ae0ce6e015042 [term 1 FOLLOWER]: Refusing update from remote peer 355e9e0b4d5e409596876e18547a020b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:42:17.258910 5190 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 355e9e0b4d5e409596876e18547a020b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3dc9bd021b146eba95ae0ce6e015042" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 33377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:42:17.266325 5036 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ccbbc4d7911f42e1bcac133c544a12da [term 1 FOLLOWER]: Refusing update from remote peer 355e9e0b4d5e409596876e18547a020b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:42:17.267567 5190 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 355e9e0b4d5e409596876e18547a020b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccbbc4d7911f42e1bcac133c544a12da" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 37529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:42:17.304220 5177 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccbbc4d7911f42e1bcac133c544a12da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 355e9e0b4d5e409596876e18547a020b. Latest consensus state: current_term: 1 leader_uuid: "355e9e0b4d5e409596876e18547a020b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355e9e0b4d5e409596876e18547a020b" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 37805 } } peers { permanent_uuid: "ccbbc4d7911f42e1bcac133c544a12da" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 37529 } } peers { permanent_uuid: "d3dc9bd021b146eba95ae0ce6e015042" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 33377 } } }
I20260925 10:42:17.304553 5177 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccbbc4d7911f42e1bcac133c544a12da [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:42:17.310101 5193 mvcc.cc:205] Tried to move back new op lower bound from 7333203710994903040 to 7333203710809681920. Current Snapshot: MvccSnapshot[applied={T|T < 7333203710994903040}]
I20260925 10:42:17.314909 5142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 355e9e0b4d5e409596876e18547a020b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "355e9e0b4d5e409596876e18547a020b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355e9e0b4d5e409596876e18547a020b" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 37805 } } peers { permanent_uuid: "ccbbc4d7911f42e1bcac133c544a12da" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 37529 } } peers { permanent_uuid: "d3dc9bd021b146eba95ae0ce6e015042" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 33377 } } }
I20260925 10:42:17.315276 5142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 355e9e0b4d5e409596876e18547a020b [sys.catalog]: This master's current role is: LEADER
I20260925 10:42:17.315826 5195 mvcc.cc:205] Tried to move back new op lower bound from 7333203710994903040 to 7333203710809681920. Current Snapshot: MvccSnapshot[applied={T|T < 7333203710994903040}]
I20260925 10:42:17.317392 5192 catalog_manager.cc:1347] Generated new cluster ID: 43c7f81ca19c4df49289512c946fb488
I20260925 10:42:17.317492 5192 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:42:17.331522 5163 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3dc9bd021b146eba95ae0ce6e015042 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 355e9e0b4d5e409596876e18547a020b. Latest consensus state: current_term: 1 leader_uuid: "355e9e0b4d5e409596876e18547a020b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355e9e0b4d5e409596876e18547a020b" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 37805 } } peers { permanent_uuid: "ccbbc4d7911f42e1bcac133c544a12da" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 37529 } } peers { permanent_uuid: "d3dc9bd021b146eba95ae0ce6e015042" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 33377 } } }
I20260925 10:42:17.331821 5163 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3dc9bd021b146eba95ae0ce6e015042 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:42:17.336625 5190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 355e9e0b4d5e409596876e18547a020b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "355e9e0b4d5e409596876e18547a020b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355e9e0b4d5e409596876e18547a020b" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 37805 } } peers { permanent_uuid: "ccbbc4d7911f42e1bcac133c544a12da" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 37529 } } peers { permanent_uuid: "d3dc9bd021b146eba95ae0ce6e015042" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 33377 } } }
I20260925 10:42:17.336897 5190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 355e9e0b4d5e409596876e18547a020b [sys.catalog]: This master's current role is: LEADER
I20260925 10:42:17.344094 5177 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccbbc4d7911f42e1bcac133c544a12da [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "355e9e0b4d5e409596876e18547a020b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355e9e0b4d5e409596876e18547a020b" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 37805 } } peers { permanent_uuid: "ccbbc4d7911f42e1bcac133c544a12da" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 37529 } } peers { permanent_uuid: "d3dc9bd021b146eba95ae0ce6e015042" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 33377 } } }
I20260925 10:42:17.344475 5177 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccbbc4d7911f42e1bcac133c544a12da [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:42:17.354756 5163 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3dc9bd021b146eba95ae0ce6e015042 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "355e9e0b4d5e409596876e18547a020b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355e9e0b4d5e409596876e18547a020b" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 37805 } } peers { permanent_uuid: "ccbbc4d7911f42e1bcac133c544a12da" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 37529 } } peers { permanent_uuid: "d3dc9bd021b146eba95ae0ce6e015042" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 33377 } } }
I20260925 10:42:17.355057 5163 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3dc9bd021b146eba95ae0ce6e015042 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:42:17.377355 5192 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:42:17.379261 5192 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:42:17.400141 5192 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 355e9e0b4d5e409596876e18547a020b: Generated new TSK 0
I20260925 10:42:17.401238 5192 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:42:17.968909 5174 catalog_manager.cc:1259] Loaded cluster ID: 43c7f81ca19c4df49289512c946fb488
I20260925 10:42:17.969029 5174 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d3dc9bd021b146eba95ae0ce6e015042: loading cluster ID for follower catalog manager: success
I20260925 10:42:17.971553 5174 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d3dc9bd021b146eba95ae0ce6e015042: acquiring CA information for follower catalog manager: success
I20260925 10:42:17.973393 5174 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d3dc9bd021b146eba95ae0ce6e015042: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 10:42:18.131261 5188 catalog_manager.cc:1259] Loaded cluster ID: 43c7f81ca19c4df49289512c946fb488
I20260925 10:42:18.131385 5188 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ccbbc4d7911f42e1bcac133c544a12da: loading cluster ID for follower catalog manager: success
I20260925 10:42:18.133718 5188 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ccbbc4d7911f42e1bcac133c544a12da: acquiring CA information for follower catalog manager: success
I20260925 10:42:18.135460 5188 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ccbbc4d7911f42e1bcac133c544a12da: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 10:42:18.490427 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 4920
I20260925 10:42:18.491643 4982 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:42:18.601495 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 4920
I20260925 10:42:18.621577 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 4994
I20260925 10:42:18.622605 5056 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:42:18.726466 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 4994
I20260925 10:42:18.746392 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 5068
I20260925 10:42:18.747463 5130 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:42:18.851706 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 5068
2026-09-25T10:42:18Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4722 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790332740062128-23952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:42:18 dist-test-slave-dxdd krb5kdc[5210](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:42:18 dist-test-slave-dxdd krb5kdc[5210](info): set up 2 sockets
Sep 25 10:42:18 dist-test-slave-dxdd krb5kdc[5210](info): commencing operation
krb5kdc: starting...
W20260925 10:42:19.934970 23952 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Sep 25 10:42:20 dist-test-slave-dxdd krb5kdc[5210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:42:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:42:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
I20260925 10:42:20.154917 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:33013
--webserver_interface=127.23.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:33755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.62:33013
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:42:20.468124 5223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:42:20.468446 5223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:42:20.468596 5223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:42:20.475682 5223 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:42:20.475814 5223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:42:20.475922 5223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:42:20.476025 5223 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:42:20.476100 5223 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:42:20.485452 5223 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:33755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790332740062128-23952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.62:33013
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:33013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:42:20.486730 5223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:42:20.487937 5223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:42:20.499763 5228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:42:20.499989 5229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:42:20.499929 5231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:42:20.500941 5223 server_base.cc:1034] running on GCE node
I20260925 10:42:20.501773 5223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:42:20.503234 5223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:42:20.504488 5223 hybrid_clock.cc:648] HybridClock initialized: now 1790332940504454 us; error 42 us; skew 500 ppm
I20260925 10:42:20.504839 5223 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:42:20 dist-test-slave-dxdd krb5kdc[5210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332940, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:42:20.517305 5223 init.cc:377] Logged in from keytab as kudu/127.23.100.62@KRBTEST.COM (short username kudu)
I20260925 10:42:20.518872 5223 webserver.cc:460] Webserver started at http://127.23.100.62:35719/ using document root <none> and password file <none>
I20260925 10:42:20.519565 5223 fs_manager.cc:359] Metadata directory not provided
I20260925 10:42:20.519672 5223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:42:20.519932 5223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:42:20.523036 5223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790332740062128-23952-0/minicluster-data/master-0/data/instance:
uuid: "b6a8618c51c9401d9ce75fef065b739e"
format_stamp: "Formatted at 2026-09-25 10:42:20 on dist-test-slave-dxdd"
server_key: "cb1c74f71cff93ff6d295150f05e1b5c"
server_key_iv: "2b48431cc4a1c57105c7ce96a7b64687"
server_key_version: "encryptionkey@0"
I20260925 10:42:20.523774 5223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790332740062128-23952-0/minicluster-data/master-0/wal/instance:
uuid: "b6a8618c51c9401d9ce75fef065b739e"
format_stamp: "Formatted at 2026-09-25 10:42:20 on dist-test-slave-dxdd"
server_key: "cb1c74f71cff93ff6d295150f05e1b5c"
server_key_iv: "2b48431cc4a1c57105c7ce96a7b64687"
server_key_version: "encryptionkey@0"
I20260925 10:42:20.528854 5223 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 10:42:20.532085 5238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:42:20.533331 5223 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260925 10:42:20.533495 5223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790332740062128-23952-0/minicluster-data/master-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790332740062128-23952-0/minicluster-data/master-0/wal
uuid: "b6a8618c51c9401d9ce75fef065b739e"
format_stamp: "Formatted at 2026-09-25 10:42:20 on dist-test-slave-dxdd"
server_key: "cb1c74f71cff93ff6d295150f05e1b5c"
server_key_iv: "2b48431cc4a1c57105c7ce96a7b64687"
server_key_version: "encryptionkey@0"
I20260925 10:42:20.533674 5223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790332740062128-23952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790332740062128-23952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790332740062128-23952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:42:20.554177 5223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:42:20.557940 5223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:42:20.558164 5223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:42:20.576949 5223 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.62:33013
I20260925 10:42:20.576956 5289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.62:33013 every 8 connection(s)
I20260925 10:42:20.578329 5223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
I20260925 10:42:20.581743 5290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:42:20.585590 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 5223
I20260925 10:42:20.585821 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790332740062128-23952-0/minicluster-data/master-0/wal/instance
I20260925 10:42:20.586717 23952 external_mini_cluster.cc:1428] Setting key e1365edd36d5b9d547037b7ada743176
I20260925 10:42:20.594158 5290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6a8618c51c9401d9ce75fef065b739e: Bootstrap starting.
I20260925 10:42:20.596982 5290 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6a8618c51c9401d9ce75fef065b739e: Neither blocks nor log segments found. Creating new log.
I20260925 10:42:20.598068 5290 log.cc:824] T 00000000000000000000000000000000 P b6a8618c51c9401d9ce75fef065b739e: Log is configured to *not* fsync() on all Append() calls
Sep 25 10:42:20 dist-test-slave-dxdd krb5kdc[5210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:42:20.600888 5290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6a8618c51c9401d9ce75fef065b739e: No bootstrap required, opened a new log
I20260925 10:42:20.607882 5293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:20.588610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41716 (local address 127.23.100.62:33013)
0925 10:42:20.589364 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:20.589376 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:20.589415 (+ 39us) server_negotiation.cc:407] Connection header received
0925 10:42:20.591022 (+ 1607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:20.591042 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:20.591435 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:20.591823 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:42:20.593307 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:20.594684 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:20.595921 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:20.596544 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:20.596741 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:20.600981 (+ 4240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:42:20.600995 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:42:20.601018 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:42:20.601050 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:42:20.604021 (+ 2971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:20.604702 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:20.604707 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:20.604712 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:20.604873 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:20.605255 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:20.605260 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:20.605264 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:20.605754 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:42:20.606076 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:20.606561 (+ 485us) server_negotiation.cc:299] Negotiation successful
0925 10:42:20.606949 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":342,"threads_started":1}
I20260925 10:42:20.608897 5290 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6a8618c51c9401d9ce75fef065b739e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6a8618c51c9401d9ce75fef065b739e" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33013 } }
I20260925 10:42:20.609230 5290 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6a8618c51c9401d9ce75fef065b739e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:42:20.609325 5290 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6a8618c51c9401d9ce75fef065b739e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6a8618c51c9401d9ce75fef065b739e, State: Initialized, Role: FOLLOWER
I20260925 10:42:20.609864 5290 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6a8618c51c9401d9ce75fef065b739e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6a8618c51c9401d9ce75fef065b739e" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33013 } }
I20260925 10:42:20.610140 5290 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6a8618c51c9401d9ce75fef065b739e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:42:20.610292 5290 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6a8618c51c9401d9ce75fef065b739e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:42:20.610549 5290 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6a8618c51c9401d9ce75fef065b739e [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:42:20.611783 23952 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:42:20.613615 5290 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6a8618c51c9401d9ce75fef065b739e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6a8618c51c9401d9ce75fef065b739e" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33013 } }
I20260925 10:42:20.614146 5290 leader_election.cc:302] T 00000000000000000000000000000000 P b6a8618c51c9401d9ce75fef065b739e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6a8618c51c9401d9ce75fef065b739e; no voters:
I20260925 10:42:20.614832 5290 leader_election.cc:288] T 00000000000000000000000000000000 P b6a8618c51c9401d9ce75fef065b739e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:42:20.614948 5295 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6a8618c51c9401d9ce75fef065b739e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:42:20.616251 5295 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6a8618c51c9401d9ce75fef065b739e [term 1 LEADER]: Becoming Leader. State: Replica: b6a8618c51c9401d9ce75fef065b739e, State: Running, Role: LEADER
I20260925 10:42:20.616711 5295 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6a8618c51c9401d9ce75fef065b739e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6a8618c51c9401d9ce75fef065b739e" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33013 } }
I20260925 10:42:20.616852 5290 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6a8618c51c9401d9ce75fef065b739e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:42:20.619630 5297 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6a8618c51c9401d9ce75fef065b739e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6a8618c51c9401d9ce75fef065b739e. Latest consensus state: current_term: 1 leader_uuid: "b6a8618c51c9401d9ce75fef065b739e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6a8618c51c9401d9ce75fef065b739e" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33013 } } }
I20260925 10:42:20.620082 5297 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6a8618c51c9401d9ce75fef065b739e [sys.catalog]: This master's current role is: LEADER
I20260925 10:42:20.620636 5296 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6a8618c51c9401d9ce75fef065b739e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6a8618c51c9401d9ce75fef065b739e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6a8618c51c9401d9ce75fef065b739e" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33013 } } }
I20260925 10:42:20.620920 5296 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6a8618c51c9401d9ce75fef065b739e [sys.catalog]: This master's current role is: LEADER
I20260925 10:42:20.623394 5304 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:42:20.627875 5304 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:42:20.635699 5304 catalog_manager.cc:1347] Generated new cluster ID: 1d60b55e21a44ff5967e3cab2808c62b
I20260925 10:42:20.635800 5304 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:42:20.653457 5304 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:42:20.654743 5304 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:42:20.665562 5304 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6a8618c51c9401d9ce75fef065b739e: Generated new TSK 0
I20260925 10:42:20.666283 5304 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:42:20.724975 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 5223
I20260925 10:42:20.726111 5285 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:42:20.821067 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 5223
2026-09-25T10:42:20Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1964 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:42:20 dist-test-slave-dxdd krb5kdc[5318](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:42:20 dist-test-slave-dxdd krb5kdc[5318](info): set up 2 sockets
Sep 25 10:42:20 dist-test-slave-dxdd krb5kdc[5318](info): commencing operation
krb5kdc: starting...
W20260925 10:42:21.902832 23952 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Sep 25 10:42:22 dist-test-slave-dxdd krb5kdc[5318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:42:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:42:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
I20260925 10:42:22.137036 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:44293
--webserver_interface=127.23.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:34751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.62:44293
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:42:22.449049 5331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:42:22.449379 5331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:42:22.449538 5331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:42:22.456400 5331 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:42:22.456528 5331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:42:22.456636 5331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:42:22.456736 5331 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:42:22.456804 5331 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:42:22.466364 5331 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:34751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.62:44293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:44293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:42:22.467609 5331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:42:22.468865 5331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:42:22.480302 5336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:42:22.480283 5337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:42:22.480388 5339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:42:22.481017 5331 server_base.cc:1034] running on GCE node
I20260925 10:42:22.481716 5331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:42:22.483091 5331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:42:22.484292 5331 hybrid_clock.cc:648] HybridClock initialized: now 1790332942484245 us; error 32 us; skew 500 ppm
I20260925 10:42:22.484622 5331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:42:22 dist-test-slave-dxdd krb5kdc[5318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332942, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:42:22.487628 5331 init.cc:377] Logged in from keytab as kudu/127.23.100.62@KRBTEST.COM (short username kudu)
I20260925 10:42:22.489044 5331 webserver.cc:460] Webserver started at http://127.23.100.62:34661/ using document root <none> and password file <none>
I20260925 10:42:22.489679 5331 fs_manager.cc:359] Metadata directory not provided
I20260925 10:42:22.489812 5331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:42:22.490114 5331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:42:22.493027 5331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/master-0/data/instance:
uuid: "120f316264344be088c2a7ab0c815611"
format_stamp: "Formatted at 2026-09-25 10:42:22 on dist-test-slave-dxdd"
server_key: "63bbc826e1c633d016bed2354f899237"
server_key_iv: "f0cf8bc7ae9cd7e2c8cec39a0d24542e"
server_key_version: "encryptionkey@0"
I20260925 10:42:22.493764 5331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/master-0/wal/instance:
uuid: "120f316264344be088c2a7ab0c815611"
format_stamp: "Formatted at 2026-09-25 10:42:22 on dist-test-slave-dxdd"
server_key: "63bbc826e1c633d016bed2354f899237"
server_key_iv: "f0cf8bc7ae9cd7e2c8cec39a0d24542e"
server_key_version: "encryptionkey@0"
I20260925 10:42:22.498752 5331 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260925 10:42:22.502066 5346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:42:22.503358 5331 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:42:22.503539 5331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/master-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/master-0/wal
uuid: "120f316264344be088c2a7ab0c815611"
format_stamp: "Formatted at 2026-09-25 10:42:22 on dist-test-slave-dxdd"
server_key: "63bbc826e1c633d016bed2354f899237"
server_key_iv: "f0cf8bc7ae9cd7e2c8cec39a0d24542e"
server_key_version: "encryptionkey@0"
I20260925 10:42:22.503732 5331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:42:22.519692 5331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:42:22.523512 5331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:42:22.523789 5331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:42:22.541401 5331 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.62:44293
I20260925 10:42:22.541424 5397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.62:44293 every 8 connection(s)
I20260925 10:42:22.542721 5331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
I20260925 10:42:22.545586 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 5331
I20260925 10:42:22.545810 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/master-0/wal/instance
I20260925 10:42:22.546646 5398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:42:22.546665 23952 external_mini_cluster.cc:1428] Setting key 4991e20ccbec19fa3c94f81f65a3b81d
I20260925 10:42:22.559804 5398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611: Bootstrap starting.
Sep 25 10:42:22 dist-test-slave-dxdd krb5kdc[5318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:42:22.562901 5398 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611: Neither blocks nor log segments found. Creating new log.
I20260925 10:42:22.564056 5398 log.cc:824] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611: Log is configured to *not* fsync() on all Append() calls
I20260925 10:42:22.566725 5398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611: No bootstrap required, opened a new log
I20260925 10:42:22.568420 5401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:22.548565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52924 (local address 127.23.100.62:44293)
0925 10:42:22.549312 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:22.549326 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:22.549363 (+ 37us) server_negotiation.cc:407] Connection header received
0925 10:42:22.551230 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:22.551249 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:22.551584 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:22.551887 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:42:22.553526 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:22.554815 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:22.556609 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:22.557280 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:22.557510 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:22.561621 (+ 4111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:42:22.561629 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:42:22.561641 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:42:22.561676 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:42:22.564702 (+ 3026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:22.565351 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:22.565357 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:22.565362 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:22.565515 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:22.565877 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:22.565882 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:22.565889 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:22.566437 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:42:22.566747 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:22.567171 (+ 424us) server_negotiation.cc:299] Negotiation successful
0925 10:42:22.567531 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":340,"threads_started":1}
I20260925 10:42:22.574175 5398 raft_consensus.cc:348] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "120f316264344be088c2a7ab0c815611" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 44293 } }
I20260925 10:42:22.574594 5398 raft_consensus.cc:374] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:42:22.574728 5398 raft_consensus.cc:729] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 120f316264344be088c2a7ab0c815611, State: Initialized, Role: FOLLOWER
I20260925 10:42:22.575289 5398 consensus_queue.cc:260] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "120f316264344be088c2a7ab0c815611" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 44293 } }
I20260925 10:42:22.575508 5398 raft_consensus.cc:388] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:42:22.575634 5398 raft_consensus.cc:482] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:42:22.575793 5398 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:42:22.577754 5398 raft_consensus.cc:504] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "120f316264344be088c2a7ab0c815611" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 44293 } }
I20260925 10:42:22.578258 5398 leader_election.cc:302] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 120f316264344be088c2a7ab0c815611; no voters:
I20260925 10:42:22.578949 5398 leader_election.cc:288] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:42:22.579061 5403 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:42:22.579883 5403 raft_consensus.cc:686] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611 [term 1 LEADER]: Becoming Leader. State: Replica: 120f316264344be088c2a7ab0c815611, State: Running, Role: LEADER
I20260925 10:42:22.580401 5403 consensus_queue.cc:237] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "120f316264344be088c2a7ab0c815611" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 44293 } }
I20260925 10:42:22.580899 5398 sys_catalog.cc:564] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:42:22.583801 5405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 120f316264344be088c2a7ab0c815611. Latest consensus state: current_term: 1 leader_uuid: "120f316264344be088c2a7ab0c815611" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "120f316264344be088c2a7ab0c815611" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 44293 } } }
I20260925 10:42:22.584231 5405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611 [sys.catalog]: This master's current role is: LEADER
I20260925 10:42:22.589819 5404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "120f316264344be088c2a7ab0c815611" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "120f316264344be088c2a7ab0c815611" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 44293 } } }
I20260925 10:42:22.590512 5404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611 [sys.catalog]: This master's current role is: LEADER
I20260925 10:42:22.590688 5410 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:42:22.595628 5410 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:42:22.604957 5410 catalog_manager.cc:1347] Generated new cluster ID: e67f1f3cc2224ce1903cb5b534b8f813
I20260925 10:42:22.605075 5410 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:42:22.631083 5410 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:42:22.632342 5410 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:42:22.638101 5410 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611: Generated new TSK 0
I20260925 10:42:22.639087 5410 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab.
Entry for principal kudu/127.23.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab.
I20260925 10:42:22.697196 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.1:0
--local_ip_for_outbound_sockets=127.23.100.1
--webserver_interface=127.23.100.1
--webserver_port=0
--tserver_master_addrs=127.23.100.62:44293
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab
--principal=kudu/127.23.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:34751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:42:23.012815 5424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:42:23.013070 5424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:42:23.013211 5424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:42:23.020191 5424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:42:23.020288 5424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:42:23.020457 5424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.1
I20260925 10:42:23.029747 5424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:34751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab
--principal=kudu/127.23.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.1
--webserver_port=0
--tserver_master_addrs=127.23.100.62:44293
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.1
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:42:23.030906 5424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:42:23.032117 5424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:42:23.044582 5432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:42:23.044649 5429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:42:23.044627 5430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:42:23.045507 5424 server_base.cc:1034] running on GCE node
I20260925 10:42:23.046115 5424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:42:23.047441 5424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:42:23.048700 5424 hybrid_clock.cc:648] HybridClock initialized: now 1790332943048616 us; error 50 us; skew 500 ppm
I20260925 10:42:23.049021 5424 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:42:23 dist-test-slave-dxdd krb5kdc[5318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332943, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:42:23.053400 5424 init.cc:377] Logged in from keytab as kudu/127.23.100.1@KRBTEST.COM (short username kudu)
I20260925 10:42:23.055164 5424 webserver.cc:460] Webserver started at http://127.23.100.1:43109/ using document root <none> and password file <none>
I20260925 10:42:23.055804 5424 fs_manager.cc:359] Metadata directory not provided
I20260925 10:42:23.055913 5424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:42:23.056154 5424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:42:23.058844 5424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-0/data/instance:
uuid: "fbac24feb0f74a2583264578c3dcf2b3"
format_stamp: "Formatted at 2026-09-25 10:42:23 on dist-test-slave-dxdd"
server_key: "1e916f473bc49b2cacaf9185d2fc9817"
server_key_iv: "f91717781fb3e3deee175992a8ce98dc"
server_key_version: "encryptionkey@0"
I20260925 10:42:23.059525 5424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-0/wal/instance:
uuid: "fbac24feb0f74a2583264578c3dcf2b3"
format_stamp: "Formatted at 2026-09-25 10:42:23 on dist-test-slave-dxdd"
server_key: "1e916f473bc49b2cacaf9185d2fc9817"
server_key_iv: "f91717781fb3e3deee175992a8ce98dc"
server_key_version: "encryptionkey@0"
I20260925 10:42:23.064128 5424 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 10:42:23.067317 5439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:42:23.068614 5424 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 10:42:23.068768 5424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-0/wal
uuid: "fbac24feb0f74a2583264578c3dcf2b3"
format_stamp: "Formatted at 2026-09-25 10:42:23 on dist-test-slave-dxdd"
server_key: "1e916f473bc49b2cacaf9185d2fc9817"
server_key_iv: "f91717781fb3e3deee175992a8ce98dc"
server_key_version: "encryptionkey@0"
I20260925 10:42:23.068960 5424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:42:23.105120 5424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:42:23.109310 5424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:42:23.109566 5424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:42:23.110527 5424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:42:23.111725 5424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:42:23.111797 5424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:42:23.111951 5424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:42:23.112005 5424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:42:23.143560 5424 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.1:36355
I20260925 10:42:23.143644 5551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.1:36355 every 8 connection(s)
I20260925 10:42:23.144814 5424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
I20260925 10:42:23.147269 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 5424
I20260925 10:42:23.147518 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-0/wal/instance
I20260925 10:42:23.148275 23952 external_mini_cluster.cc:1428] Setting key 34bb456d11eeb1068685bbaff8d6b23d
Sep 25 10:42:23 dist-test-slave-dxdd krb5kdc[5318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332943, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.1@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:42:23.164444 5554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:23.147318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:51473 (local address 127.23.100.62:44293)
0925 10:42:23.147853 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:23.147860 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:23.148902 (+ 1042us) server_negotiation.cc:407] Connection header received
0925 10:42:23.150997 (+ 2095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:23.151002 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:23.151090 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:23.151204 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:42:23.152636 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:23.153354 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:23.154975 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:23.155390 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:23.155526 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:23.158944 (+ 3418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:42:23.158949 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:42:23.158952 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:42:23.158982 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:42:23.161498 (+ 2516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:23.162179 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:23.162188 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:23.162190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:23.162298 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:23.162782 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:23.162787 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:23.162790 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:23.163044 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:42:23.163198 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:23.163848 (+ 650us) server_negotiation.cc:299] Negotiation successful
0925 10:42:23.164060 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":245,"threads_started":1}
I20260925 10:42:23.165465 5555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:23.147902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.1:51473)
0925 10:42:23.148729 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:23.148780 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 10:42:23.150739 (+ 1959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:42:23.151442 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:42:23.151451 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:42:23.151929 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:42:23.152453 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:23.152474 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:23.153508 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:23.153514 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:23.154790 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:23.154801 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:23.155629 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:23.155636 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:23.155859 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:23.156562 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:42:23.156584 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:42:23.158745 (+ 2161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:42:23.161630 (+ 2885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:42:23.161639 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:42:23.161653 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:42:23.162056 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:42:23.162447 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:42:23.162455 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:42:23.162459 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:42:23.162666 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:42:23.163170 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:42:23.163178 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:42:23.163497 (+ 319us) client_negotiation.cc:769] Sending connection context
0925 10:42:23.163738 (+ 241us) client_negotiation.cc:240] Negotiation successful
0925 10:42:23.164250 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":327,"threads_started":1}
I20260925 10:42:23.167043 5552 heartbeater.cc:347] Connected to a master server at 127.23.100.62:44293
I20260925 10:42:23.167384 5552 heartbeater.cc:464] Registering TS with master...
I20260925 10:42:23.168298 5552 heartbeater.cc:511] Master 127.23.100.62:44293 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.100.2@KRBTEST.COM; defaulting to no policy
I20260925 10:42:23.170607 5363 ts_manager.cc:194] Registered new tserver with Master: fbac24feb0f74a2583264578c3dcf2b3 (127.23.100.1:36355)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.2@KRBTEST.COM" created.
I20260925 10:42:23.172279 5363 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.1@KRBTEST.COM'} at 127.23.100.1:51473
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab.
Entry for principal kudu/127.23.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab.
I20260925 10:42:23.198097 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.2:0
--local_ip_for_outbound_sockets=127.23.100.2
--webserver_interface=127.23.100.2
--webserver_port=0
--tserver_master_addrs=127.23.100.62:44293
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab
--principal=kudu/127.23.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:34751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:42:23.511227 5558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:42:23.511464 5558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:42:23.511662 5558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:42:23.518636 5558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:42:23.518725 5558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:42:23.518921 5558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.2
I20260925 10:42:23.528548 5558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:34751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab
--principal=kudu/127.23.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.2
--webserver_port=0
--tserver_master_addrs=127.23.100.62:44293
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.2
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:42:23.529675 5558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:42:23.530942 5558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:42:23.543938 5564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:42:23.544100 5566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:42:23.544400 5563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:42:23.544950 5558 server_base.cc:1034] running on GCE node
I20260925 10:42:23.545506 5558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:42:23.546396 5558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:42:23.547607 5558 hybrid_clock.cc:648] HybridClock initialized: now 1790332943547557 us; error 44 us; skew 500 ppm
I20260925 10:42:23.547892 5558 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:42:23 dist-test-slave-dxdd krb5kdc[5318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332943, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:42:23.550915 5558 init.cc:377] Logged in from keytab as kudu/127.23.100.2@KRBTEST.COM (short username kudu)
I20260925 10:42:23.552340 5558 webserver.cc:460] Webserver started at http://127.23.100.2:42147/ using document root <none> and password file <none>
I20260925 10:42:23.552955 5558 fs_manager.cc:359] Metadata directory not provided
I20260925 10:42:23.553102 5558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:42:23.553395 5558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:42:23.556057 5558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-1/data/instance:
uuid: "899b0712f9d24f4a9a2e0d518962620d"
format_stamp: "Formatted at 2026-09-25 10:42:23 on dist-test-slave-dxdd"
server_key: "befdc78e3b4c2dcb2ed0d22bd73f1efa"
server_key_iv: "2533722dd23d7c65d40a21e0e1c9e344"
server_key_version: "encryptionkey@0"
I20260925 10:42:23.556782 5558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-1/wal/instance:
uuid: "899b0712f9d24f4a9a2e0d518962620d"
format_stamp: "Formatted at 2026-09-25 10:42:23 on dist-test-slave-dxdd"
server_key: "befdc78e3b4c2dcb2ed0d22bd73f1efa"
server_key_iv: "2533722dd23d7c65d40a21e0e1c9e344"
server_key_version: "encryptionkey@0"
I20260925 10:42:23.561542 5558 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 10:42:23.564658 5573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:42:23.566010 5558 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:42:23.566177 5558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-1/wal
uuid: "899b0712f9d24f4a9a2e0d518962620d"
format_stamp: "Formatted at 2026-09-25 10:42:23 on dist-test-slave-dxdd"
server_key: "befdc78e3b4c2dcb2ed0d22bd73f1efa"
server_key_iv: "2533722dd23d7c65d40a21e0e1c9e344"
server_key_version: "encryptionkey@0"
I20260925 10:42:23.566375 5558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:42:23.591794 5558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:42:23.595643 5558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:42:23.595924 5558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:42:23.596819 5558 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:42:23.598094 5558 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:42:23.598193 5558 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:42:23.598358 5558 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:42:23.598433 5558 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260925 10:42:23.630651 5558 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.2:34201
I20260925 10:42:23.630728 5685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.2:34201 every 8 connection(s)
I20260925 10:42:23.631846 5558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
I20260925 10:42:23.638782 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 5558
I20260925 10:42:23.638970 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-1/wal/instance
I20260925 10:42:23.639446 23952 external_mini_cluster.cc:1428] Setting key 94d7eda4116607e104faf801fd1534d0
Sep 25 10:42:23 dist-test-slave-dxdd krb5kdc[5318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332943, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.2@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:42:23.652906 5554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:23.634180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:47455 (local address 127.23.100.62:44293)
0925 10:42:23.634407 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:23.634411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:23.635417 (+ 1006us) server_negotiation.cc:407] Connection header received
0925 10:42:23.637146 (+ 1729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:23.637151 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:23.637242 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:23.637352 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:42:23.639044 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:23.639900 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:23.641894 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:23.642650 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:23.642794 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:23.646175 (+ 3381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:42:23.646184 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:42:23.646192 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:42:23.646236 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:42:23.649783 (+ 3547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:23.650727 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:23.650731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:23.650733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:23.650877 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:23.651327 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:23.651330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:23.651332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:23.651581 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:42:23.651688 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:23.652487 (+ 799us) server_negotiation.cc:299] Negotiation successful
0925 10:42:23.652667 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 10:42:23.654346 5688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:23.634607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.2:47455)
0925 10:42:23.635270 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:23.635311 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 10:42:23.636869 (+ 1558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:42:23.637572 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:42:23.637581 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:42:23.638039 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:42:23.638797 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:23.638823 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:23.640055 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:23.640060 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:23.641751 (+ 1691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:23.641766 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:23.642837 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:23.642841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:23.643022 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:23.643668 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:42:23.643688 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:42:23.645962 (+ 2274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:42:23.649922 (+ 3960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:42:23.649933 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:42:23.649948 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:42:23.650615 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:42:23.650985 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:42:23.650993 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:42:23.650999 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:42:23.651221 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:42:23.651698 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:42:23.651708 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:42:23.652114 (+ 406us) client_negotiation.cc:769] Sending connection context
0925 10:42:23.652385 (+ 271us) client_negotiation.cc:240] Negotiation successful
0925 10:42:23.652925 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":264,"threads_started":1}
I20260925 10:42:23.656083 5686 heartbeater.cc:347] Connected to a master server at 127.23.100.62:44293
I20260925 10:42:23.656376 5686 heartbeater.cc:464] Registering TS with master...
I20260925 10:42:23.657199 5686 heartbeater.cc:511] Master 127.23.100.62:44293 requested a full tablet report, sending...
I20260925 10:42:23.658766 5363 ts_manager.cc:194] Registered new tserver with Master: 899b0712f9d24f4a9a2e0d518962620d (127.23.100.2:34201)
I20260925 10:42:23.659587 5363 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.2@KRBTEST.COM'} at 127.23.100.2:47455
WARNING: no policy specified for kudu/127.23.100.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab.
Entry for principal kudu/127.23.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab.
I20260925 10:42:23.692787 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.3:0
--local_ip_for_outbound_sockets=127.23.100.3
--webserver_interface=127.23.100.3
--webserver_port=0
--tserver_master_addrs=127.23.100.62:44293
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab
--principal=kudu/127.23.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:34751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:42:24.000877 5691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:42:24.001199 5691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:42:24.001369 5691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:42:24.008653 5691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:42:24.008787 5691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:42:24.008965 5691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.3
I20260925 10:42:24.017974 5691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:34751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab
--principal=kudu/127.23.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.3
--webserver_port=0
--tserver_master_addrs=127.23.100.62:44293
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.3
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:42:24.019377 5691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:42:24.020659 5691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:42:24.033052 5699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:42:24.033088 5697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:42:24.033530 5696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:42:24.033891 5691 server_base.cc:1034] running on GCE node
I20260925 10:42:24.034557 5691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:42:24.035444 5691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:42:24.036674 5691 hybrid_clock.cc:648] HybridClock initialized: now 1790332944036628 us; error 57 us; skew 500 ppm
I20260925 10:42:24.036986 5691 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:42:24 dist-test-slave-dxdd krb5kdc[5318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332944, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:42:24.040141 5691 init.cc:377] Logged in from keytab as kudu/127.23.100.3@KRBTEST.COM (short username kudu)
I20260925 10:42:24.041692 5691 webserver.cc:460] Webserver started at http://127.23.100.3:37913/ using document root <none> and password file <none>
I20260925 10:42:24.042384 5691 fs_manager.cc:359] Metadata directory not provided
I20260925 10:42:24.042490 5691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:42:24.042744 5691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:42:24.045675 5691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-2/data/instance:
uuid: "b41fee5f629349268798ff60f57a31bb"
format_stamp: "Formatted at 2026-09-25 10:42:24 on dist-test-slave-dxdd"
server_key: "95b3064ccfbe0b6d74a36075a4017263"
server_key_iv: "216d497b9da00f220a1e90fbbf32cc30"
server_key_version: "encryptionkey@0"
I20260925 10:42:24.046487 5691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-2/wal/instance:
uuid: "b41fee5f629349268798ff60f57a31bb"
format_stamp: "Formatted at 2026-09-25 10:42:24 on dist-test-slave-dxdd"
server_key: "95b3064ccfbe0b6d74a36075a4017263"
server_key_iv: "216d497b9da00f220a1e90fbbf32cc30"
server_key_version: "encryptionkey@0"
I20260925 10:42:24.051486 5691 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 10:42:24.054713 5706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:42:24.056003 5691 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260925 10:42:24.056172 5691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-2/wal
uuid: "b41fee5f629349268798ff60f57a31bb"
format_stamp: "Formatted at 2026-09-25 10:42:24 on dist-test-slave-dxdd"
server_key: "95b3064ccfbe0b6d74a36075a4017263"
server_key_iv: "216d497b9da00f220a1e90fbbf32cc30"
server_key_version: "encryptionkey@0"
I20260925 10:42:24.056365 5691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:42:24.080912 5691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:42:24.084705 5691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:42:24.084981 5691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:42:24.085870 5691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:42:24.087141 5691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:42:24.087210 5691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:42:24.087364 5691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:42:24.087412 5691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:42:24.119323 5691 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.3:46473
I20260925 10:42:24.119388 5818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.3:46473 every 8 connection(s)
I20260925 10:42:24.120564 5691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
I20260925 10:42:24.122895 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 5691
I20260925 10:42:24.123096 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-2/wal/instance
I20260925 10:42:24.123607 23952 external_mini_cluster.cc:1428] Setting key bf992c66e59421475e894a5f8e2b5849
Sep 25 10:42:24 dist-test-slave-dxdd krb5kdc[5318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332944, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.3@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:42:24.139395 5554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:24.123114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:41927 (local address 127.23.100.62:44293)
0925 10:42:24.123320 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:24.123326 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:24.124729 (+ 1403us) server_negotiation.cc:407] Connection header received
0925 10:42:24.126488 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:24.126493 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:24.126607 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:24.126725 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:42:24.128130 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:24.128840 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:24.130339 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:24.130746 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:24.130878 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:24.134082 (+ 3204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:42:24.134088 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:42:24.134093 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:42:24.134132 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:42:24.136473 (+ 2341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:24.137129 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:24.137134 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:24.137138 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:24.137295 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:24.137707 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:24.137713 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:24.137716 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:24.137969 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:42:24.138133 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:24.138823 (+ 690us) server_negotiation.cc:299] Negotiation successful
0925 10:42:24.139047 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 10:42:24.140235 5821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:24.123478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.3:41927)
0925 10:42:24.124550 (+ 1072us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:24.124615 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 10:42:24.126206 (+ 1591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:42:24.126960 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:42:24.126968 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:42:24.127424 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:42:24.127966 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:24.127982 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:24.128983 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:24.128988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:24.130175 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:24.130186 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:24.130955 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:24.130963 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:24.131219 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:24.131777 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:42:24.131797 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:42:24.133874 (+ 2077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:42:24.136606 (+ 2732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:42:24.136614 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:42:24.136628 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:42:24.137010 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:42:24.137405 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:42:24.137411 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:42:24.137414 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:42:24.137597 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:42:24.138109 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:42:24.138116 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:42:24.138473 (+ 357us) client_negotiation.cc:769] Sending connection context
0925 10:42:24.138730 (+ 257us) client_negotiation.cc:240] Negotiation successful
0925 10:42:24.139206 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":779,"thread_start_us":604,"threads_started":1}
I20260925 10:42:24.141436 5819 heartbeater.cc:347] Connected to a master server at 127.23.100.62:44293
I20260925 10:42:24.141728 5819 heartbeater.cc:464] Registering TS with master...
I20260925 10:42:24.142462 5819 heartbeater.cc:511] Master 127.23.100.62:44293 requested a full tablet report, sending...
I20260925 10:42:24.143900 5363 ts_manager.cc:194] Registered new tserver with Master: b41fee5f629349268798ff60f57a31bb (127.23.100.3:46473)
I20260925 10:42:24.145015 5363 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.3@KRBTEST.COM'} at 127.23.100.3:41927
I20260925 10:42:24.150290 23952 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:42:24.166905 5554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:24.154576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52934 (local address 127.23.100.62:44293)
0925 10:42:24.154768 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:24.154776 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:24.155260 (+ 484us) server_negotiation.cc:407] Connection header received
0925 10:42:24.155387 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:24.155391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:24.155446 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:24.155522 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:42:24.156864 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:24.157583 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:24.158885 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:24.159269 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:24.159370 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:24.161761 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:42:24.161766 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:42:24.161769 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:42:24.161797 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:42:24.164099 (+ 2302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:24.164753 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:24.164760 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:24.164763 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:24.164890 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:24.165296 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:24.165301 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:24.165306 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:24.165603 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:42:24.165876 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:24.166185 (+ 309us) server_negotiation.cc:299] Negotiation successful
0925 10:42:24.166578 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 10:42:24.175058 5552 heartbeater.cc:503] Master 127.23.100.62:44293 was elected leader, sending a full tablet report...
I20260925 10:42:24.170738 5363 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52934:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:42:24.178350 5363 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:42:24.213989 5833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:24.189638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.3:46473 (local address 127.0.0.1:41226)
0925 10:42:24.202524 (+ 12886us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:24.202569 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 10:42:24.202767 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:42:24.205224 (+ 2457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:42:24.205236 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:42:24.205277 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:42:24.205793 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:24.205808 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:24.208646 (+ 2838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:24.208651 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:24.210997 (+ 2346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:24.211010 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:24.213045 (+ 2035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:24.213053 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:24.213237 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:24.213288 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 10:42:24.213420 (+ 132us) client_negotiation.cc:240] Negotiation successful
0925 10:42:24.213589 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12564,"mutex_wait_us":2,"threads_started":1}
I20260925 10:42:24.214962 5834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:24.190223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41226 (local address 127.23.100.3:46473)
0925 10:42:24.204122 (+ 13899us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:24.204136 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:24.204178 (+ 42us) server_negotiation.cc:407] Connection header received
0925 10:42:24.204296 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:24.204308 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:24.204620 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:24.204842 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:42:24.206777 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:24.208467 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:24.211187 (+ 2720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:24.212556 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:24.212703 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:24.212925 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:24.214495 (+ 1570us) server_negotiation.cc:299] Negotiation successful
0925 10:42:24.214665 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13666,"thread_start_us":13519,"threads_started":1}
I20260925 10:42:24.215408 5832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:24.189880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.2:34201 (local address 127.0.0.1:41618)
0925 10:42:24.206433 (+ 16553us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:24.206453 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:42:24.206564 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:42:24.207632 (+ 1068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:42:24.207637 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:42:24.207660 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:42:24.208064 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:24.208075 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:24.209894 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:24.209899 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:24.211986 (+ 2087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:24.212001 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:24.214046 (+ 2045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:24.214055 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:24.214515 (+ 460us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:24.214544 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 10:42:24.214629 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 10:42:24.214737 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16382,"mutex_wait_us":4,"thread_start_us":16268,"threads_started":1}
I20260925 10:42:24.215998 5836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:24.190855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41618 (local address 127.23.100.2:34201)
0925 10:42:24.206777 (+ 15922us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:24.206785 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:24.206809 (+ 24us) server_negotiation.cc:407] Connection header received
0925 10:42:24.206899 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:24.206906 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:24.207122 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:24.207321 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:42:24.208250 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:24.209747 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:24.212162 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:24.213575 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:24.213707 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:24.213896 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:24.215523 (+ 1627us) server_negotiation.cc:299] Negotiation successful
0925 10:42:24.215692 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15762,"thread_start_us":15619,"threads_started":1}
I20260925 10:42:24.219304 5621 tablet_service.cc:1467] Processing CreateTablet for tablet ea42676e7b2e441087764f3874cfe8f5 (DEFAULT_TABLE table=test-table [id=4a646f4b415c46fc8a6118d0f37ddf0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:42:24.219286 5754 tablet_service.cc:1467] Processing CreateTablet for tablet ea42676e7b2e441087764f3874cfe8f5 (DEFAULT_TABLE table=test-table [id=4a646f4b415c46fc8a6118d0f37ddf0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:42:24.220479 5621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea42676e7b2e441087764f3874cfe8f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:42:24.220625 5754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea42676e7b2e441087764f3874cfe8f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:42:24.221529 5833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:24.190010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:36355 (local address 127.0.0.1:45468)
0925 10:42:24.214224 (+ 24214us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:24.214239 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:42:24.214343 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:42:24.215277 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:42:24.215281 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:42:24.215307 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:42:24.215478 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:24.215490 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:24.217060 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:24.217078 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:24.218899 (+ 1821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:24.218914 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:24.220830 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:24.220836 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:24.221024 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:24.221064 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 10:42:24.221161 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 10:42:24.221282 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24137,"thread_start_us":12953,"threads_started":1}
I20260925 10:42:24.221712 5835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:24.190377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45468 (local address 127.23.100.1:36355)
0925 10:42:24.214114 (+ 23737us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:24.214125 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:24.214371 (+ 246us) server_negotiation.cc:407] Connection header received
0925 10:42:24.214502 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:24.214511 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:24.214754 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:24.214972 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:42:24.215666 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:24.216928 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:24.219088 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:24.220538 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:24.220680 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:24.220880 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:24.221232 (+ 352us) server_negotiation.cc:299] Negotiation successful
0925 10:42:24.221378 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23507,"thread_start_us":23732,"threads_started":1}
I20260925 10:42:24.225103 5487 tablet_service.cc:1467] Processing CreateTablet for tablet ea42676e7b2e441087764f3874cfe8f5 (DEFAULT_TABLE table=test-table [id=4a646f4b415c46fc8a6118d0f37ddf0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:42:24.226424 5487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea42676e7b2e441087764f3874cfe8f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:42:24.231359 5837 tablet_bootstrap.cc:492] T ea42676e7b2e441087764f3874cfe8f5 P b41fee5f629349268798ff60f57a31bb: Bootstrap starting.
I20260925 10:42:24.234441 5838 tablet_bootstrap.cc:492] T ea42676e7b2e441087764f3874cfe8f5 P 899b0712f9d24f4a9a2e0d518962620d: Bootstrap starting.
I20260925 10:42:24.234467 5837 tablet_bootstrap.cc:654] T ea42676e7b2e441087764f3874cfe8f5 P b41fee5f629349268798ff60f57a31bb: Neither blocks nor log segments found. Creating new log.
I20260925 10:42:24.235749 5837 log.cc:824] T ea42676e7b2e441087764f3874cfe8f5 P b41fee5f629349268798ff60f57a31bb: Log is configured to *not* fsync() on all Append() calls
I20260925 10:42:24.237700 5838 tablet_bootstrap.cc:654] T ea42676e7b2e441087764f3874cfe8f5 P 899b0712f9d24f4a9a2e0d518962620d: Neither blocks nor log segments found. Creating new log.
I20260925 10:42:24.238916 5837 tablet_bootstrap.cc:492] T ea42676e7b2e441087764f3874cfe8f5 P b41fee5f629349268798ff60f57a31bb: No bootstrap required, opened a new log
I20260925 10:42:24.238902 5838 log.cc:824] T ea42676e7b2e441087764f3874cfe8f5 P 899b0712f9d24f4a9a2e0d518962620d: Log is configured to *not* fsync() on all Append() calls
I20260925 10:42:24.239182 5837 ts_tablet_manager.cc:1397] T ea42676e7b2e441087764f3874cfe8f5 P b41fee5f629349268798ff60f57a31bb: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260925 10:42:24.240731 5841 tablet_bootstrap.cc:492] T ea42676e7b2e441087764f3874cfe8f5 P fbac24feb0f74a2583264578c3dcf2b3: Bootstrap starting.
I20260925 10:42:24.241652 5838 tablet_bootstrap.cc:492] T ea42676e7b2e441087764f3874cfe8f5 P 899b0712f9d24f4a9a2e0d518962620d: No bootstrap required, opened a new log
I20260925 10:42:24.241882 5838 ts_tablet_manager.cc:1397] T ea42676e7b2e441087764f3874cfe8f5 P 899b0712f9d24f4a9a2e0d518962620d: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 10:42:24.243655 5841 tablet_bootstrap.cc:654] T ea42676e7b2e441087764f3874cfe8f5 P fbac24feb0f74a2583264578c3dcf2b3: Neither blocks nor log segments found. Creating new log.
I20260925 10:42:24.244541 5841 log.cc:824] T ea42676e7b2e441087764f3874cfe8f5 P fbac24feb0f74a2583264578c3dcf2b3: Log is configured to *not* fsync() on all Append() calls
I20260925 10:42:24.246898 5841 tablet_bootstrap.cc:492] T ea42676e7b2e441087764f3874cfe8f5 P fbac24feb0f74a2583264578c3dcf2b3: No bootstrap required, opened a new log
I20260925 10:42:24.247138 5841 ts_tablet_manager.cc:1397] T ea42676e7b2e441087764f3874cfe8f5 P fbac24feb0f74a2583264578c3dcf2b3: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s
I20260925 10:42:24.249542 5837 raft_consensus.cc:348] T ea42676e7b2e441087764f3874cfe8f5 P b41fee5f629349268798ff60f57a31bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b41fee5f629349268798ff60f57a31bb" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 46473 } } peers { permanent_uuid: "899b0712f9d24f4a9a2e0d518962620d" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 34201 } } peers { permanent_uuid: "fbac24feb0f74a2583264578c3dcf2b3" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 36355 } }
I20260925 10:42:24.249953 5837 raft_consensus.cc:374] T ea42676e7b2e441087764f3874cfe8f5 P b41fee5f629349268798ff60f57a31bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:42:24.250044 5837 raft_consensus.cc:729] T ea42676e7b2e441087764f3874cfe8f5 P b41fee5f629349268798ff60f57a31bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b41fee5f629349268798ff60f57a31bb, State: Initialized, Role: FOLLOWER
I20260925 10:42:24.250669 5837 consensus_queue.cc:260] T ea42676e7b2e441087764f3874cfe8f5 P b41fee5f629349268798ff60f57a31bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b41fee5f629349268798ff60f57a31bb" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 46473 } } peers { permanent_uuid: "899b0712f9d24f4a9a2e0d518962620d" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 34201 } } peers { permanent_uuid: "fbac24feb0f74a2583264578c3dcf2b3" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 36355 } }
I20260925 10:42:24.252209 5819 heartbeater.cc:503] Master 127.23.100.62:44293 was elected leader, sending a full tablet report...
I20260925 10:42:24.252417 5838 raft_consensus.cc:348] T ea42676e7b2e441087764f3874cfe8f5 P 899b0712f9d24f4a9a2e0d518962620d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b41fee5f629349268798ff60f57a31bb" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 46473 } } peers { permanent_uuid: "899b0712f9d24f4a9a2e0d518962620d" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 34201 } } peers { permanent_uuid: "fbac24feb0f74a2583264578c3dcf2b3" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 36355 } }
I20260925 10:42:24.252841 5838 raft_consensus.cc:374] T ea42676e7b2e441087764f3874cfe8f5 P 899b0712f9d24f4a9a2e0d518962620d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:42:24.252935 5837 ts_tablet_manager.cc:1428] T ea42676e7b2e441087764f3874cfe8f5 P b41fee5f629349268798ff60f57a31bb: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260925 10:42:24.252935 5838 raft_consensus.cc:729] T ea42676e7b2e441087764f3874cfe8f5 P 899b0712f9d24f4a9a2e0d518962620d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 899b0712f9d24f4a9a2e0d518962620d, State: Initialized, Role: FOLLOWER
I20260925 10:42:24.253592 5838 consensus_queue.cc:260] T ea42676e7b2e441087764f3874cfe8f5 P 899b0712f9d24f4a9a2e0d518962620d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b41fee5f629349268798ff60f57a31bb" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 46473 } } peers { permanent_uuid: "899b0712f9d24f4a9a2e0d518962620d" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 34201 } } peers { permanent_uuid: "fbac24feb0f74a2583264578c3dcf2b3" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 36355 } }
I20260925 10:42:24.254645 5686 heartbeater.cc:503] Master 127.23.100.62:44293 was elected leader, sending a full tablet report...
I20260925 10:42:24.255287 5838 ts_tablet_manager.cc:1428] T ea42676e7b2e441087764f3874cfe8f5 P 899b0712f9d24f4a9a2e0d518962620d: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260925 10:42:24.255713 5841 raft_consensus.cc:348] T ea42676e7b2e441087764f3874cfe8f5 P fbac24feb0f74a2583264578c3dcf2b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b41fee5f629349268798ff60f57a31bb" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 46473 } } peers { permanent_uuid: "899b0712f9d24f4a9a2e0d518962620d" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 34201 } } peers { permanent_uuid: "fbac24feb0f74a2583264578c3dcf2b3" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 36355 } }
I20260925 10:42:24.256104 5841 raft_consensus.cc:374] T ea42676e7b2e441087764f3874cfe8f5 P fbac24feb0f74a2583264578c3dcf2b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:42:24.256213 5841 raft_consensus.cc:729] T ea42676e7b2e441087764f3874cfe8f5 P fbac24feb0f74a2583264578c3dcf2b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbac24feb0f74a2583264578c3dcf2b3, State: Initialized, Role: FOLLOWER
I20260925 10:42:24.256804 5841 consensus_queue.cc:260] T ea42676e7b2e441087764f3874cfe8f5 P fbac24feb0f74a2583264578c3dcf2b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b41fee5f629349268798ff60f57a31bb" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 46473 } } peers { permanent_uuid: "899b0712f9d24f4a9a2e0d518962620d" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 34201 } } peers { permanent_uuid: "fbac24feb0f74a2583264578c3dcf2b3" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 36355 } }
I20260925 10:42:24.259301 5841 ts_tablet_manager.cc:1428] T ea42676e7b2e441087764f3874cfe8f5 P fbac24feb0f74a2583264578c3dcf2b3: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260925 10:42:24.325557 5845 raft_consensus.cc:482] T ea42676e7b2e441087764f3874cfe8f5 P fbac24feb0f74a2583264578c3dcf2b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:42:24.325887 5845 raft_consensus.cc:504] T ea42676e7b2e441087764f3874cfe8f5 P fbac24feb0f74a2583264578c3dcf2b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b41fee5f629349268798ff60f57a31bb" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 46473 } } peers { permanent_uuid: "899b0712f9d24f4a9a2e0d518962620d" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 34201 } } peers { permanent_uuid: "fbac24feb0f74a2583264578c3dcf2b3" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 36355 } }
I20260925 10:42:24.327596 5845 leader_election.cc:288] T ea42676e7b2e441087764f3874cfe8f5 P fbac24feb0f74a2583264578c3dcf2b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b41fee5f629349268798ff60f57a31bb (127.23.100.3:46473), 899b0712f9d24f4a9a2e0d518962620d (127.23.100.2:34201)
I20260925 10:42:24.334014 5836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:24.328029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:48533 (local address 127.23.100.2:34201)
0925 10:42:24.328331 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:24.328339 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:24.328829 (+ 490us) server_negotiation.cc:407] Connection header received
0925 10:42:24.329063 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:24.329072 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:24.329130 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:24.329228 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:42:24.329780 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:24.330554 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:24.332287 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:24.333051 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:24.333222 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:24.333295 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:24.333637 (+ 342us) server_negotiation.cc:299] Negotiation successful
0925 10:42:24.333757 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260925 10:42:24.334110 5834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:24.327898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:54539 (local address 127.23.100.3:46473)
0925 10:42:24.328107 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:24.328115 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:24.329051 (+ 936us) server_negotiation.cc:407] Connection header received
0925 10:42:24.329253 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:24.329258 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:24.329315 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:24.329414 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:42:24.329987 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:24.330746 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:24.332438 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:24.333179 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:24.333350 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:24.333426 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:24.333729 (+ 303us) server_negotiation.cc:299] Negotiation successful
0925 10:42:24.333848 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 10:42:24.334168 5846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:24.327846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.2:34201 (local address 127.23.100.1:48533)
0925 10:42:24.328712 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:24.328729 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:42:24.328911 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:42:24.329242 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:42:24.329246 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:42:24.329268 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:42:24.329635 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:24.329645 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:24.330671 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:24.330679 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:24.332135 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:24.332146 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:24.333248 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:24.333255 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:24.333476 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:24.333503 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 10:42:24.333611 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 10:42:24.333732 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":754,"mutex_wait_us":38,"thread_start_us":445,"threads_started":1}
I20260925 10:42:24.334621 5847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:24.327846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.3:46473 (local address 127.23.100.1:54539)
0925 10:42:24.328937 (+ 1091us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:24.328950 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:42:24.329104 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:42:24.329435 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:42:24.329439 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:42:24.329463 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:42:24.329835 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:24.329843 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:24.330883 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:24.330888 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:24.332283 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:24.332295 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:24.333393 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:24.333399 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:24.333576 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:24.333599 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 10:42:24.333703 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 10:42:24.333820 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":985,"mutex_wait_us":148,"thread_start_us":618,"threads_started":1}
I20260925 10:42:24.335170 5641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea42676e7b2e441087764f3874cfe8f5" candidate_uuid: "fbac24feb0f74a2583264578c3dcf2b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "899b0712f9d24f4a9a2e0d518962620d" is_pre_election: true
I20260925 10:42:24.335547 5774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea42676e7b2e441087764f3874cfe8f5" candidate_uuid: "fbac24feb0f74a2583264578c3dcf2b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b41fee5f629349268798ff60f57a31bb" is_pre_election: true
I20260925 10:42:24.335664 5641 raft_consensus.cc:2454] T ea42676e7b2e441087764f3874cfe8f5 P 899b0712f9d24f4a9a2e0d518962620d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbac24feb0f74a2583264578c3dcf2b3 in term 0.
I20260925 10:42:24.336015 5774 raft_consensus.cc:2454] T ea42676e7b2e441087764f3874cfe8f5 P b41fee5f629349268798ff60f57a31bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbac24feb0f74a2583264578c3dcf2b3 in term 0.
I20260925 10:42:24.336335 5440 leader_election.cc:302] T ea42676e7b2e441087764f3874cfe8f5 P fbac24feb0f74a2583264578c3dcf2b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 899b0712f9d24f4a9a2e0d518962620d, fbac24feb0f74a2583264578c3dcf2b3; no voters:
I20260925 10:42:24.336752 5845 raft_consensus.cc:2781] T ea42676e7b2e441087764f3874cfe8f5 P fbac24feb0f74a2583264578c3dcf2b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:42:24.336905 5845 raft_consensus.cc:482] T ea42676e7b2e441087764f3874cfe8f5 P fbac24feb0f74a2583264578c3dcf2b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:42:24.337020 5845 raft_consensus.cc:3037] T ea42676e7b2e441087764f3874cfe8f5 P fbac24feb0f74a2583264578c3dcf2b3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:42:24.339180 5845 raft_consensus.cc:504] T ea42676e7b2e441087764f3874cfe8f5 P fbac24feb0f74a2583264578c3dcf2b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b41fee5f629349268798ff60f57a31bb" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 46473 } } peers { permanent_uuid: "899b0712f9d24f4a9a2e0d518962620d" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 34201 } } peers { permanent_uuid: "fbac24feb0f74a2583264578c3dcf2b3" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 36355 } }
I20260925 10:42:24.339753 5845 leader_election.cc:288] T ea42676e7b2e441087764f3874cfe8f5 P fbac24feb0f74a2583264578c3dcf2b3 [CANDIDATE]: Term 1 election: Requested vote from peers b41fee5f629349268798ff60f57a31bb (127.23.100.3:46473), 899b0712f9d24f4a9a2e0d518962620d (127.23.100.2:34201)
I20260925 10:42:24.340149 5774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea42676e7b2e441087764f3874cfe8f5" candidate_uuid: "fbac24feb0f74a2583264578c3dcf2b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b41fee5f629349268798ff60f57a31bb"
I20260925 10:42:24.340188 5641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea42676e7b2e441087764f3874cfe8f5" candidate_uuid: "fbac24feb0f74a2583264578c3dcf2b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "899b0712f9d24f4a9a2e0d518962620d"
I20260925 10:42:24.340337 5774 raft_consensus.cc:3037] T ea42676e7b2e441087764f3874cfe8f5 P b41fee5f629349268798ff60f57a31bb [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:42:24.340380 5641 raft_consensus.cc:3037] T ea42676e7b2e441087764f3874cfe8f5 P 899b0712f9d24f4a9a2e0d518962620d [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:42:24.342613 5774 raft_consensus.cc:2454] T ea42676e7b2e441087764f3874cfe8f5 P b41fee5f629349268798ff60f57a31bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbac24feb0f74a2583264578c3dcf2b3 in term 1.
I20260925 10:42:24.342705 5641 raft_consensus.cc:2454] T ea42676e7b2e441087764f3874cfe8f5 P 899b0712f9d24f4a9a2e0d518962620d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbac24feb0f74a2583264578c3dcf2b3 in term 1.
I20260925 10:42:24.343039 5441 leader_election.cc:302] T ea42676e7b2e441087764f3874cfe8f5 P fbac24feb0f74a2583264578c3dcf2b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b41fee5f629349268798ff60f57a31bb, fbac24feb0f74a2583264578c3dcf2b3; no voters:
I20260925 10:42:24.343333 5845 raft_consensus.cc:2781] T ea42676e7b2e441087764f3874cfe8f5 P fbac24feb0f74a2583264578c3dcf2b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:42:24.343756 5845 raft_consensus.cc:686] T ea42676e7b2e441087764f3874cfe8f5 P fbac24feb0f74a2583264578c3dcf2b3 [term 1 LEADER]: Becoming Leader. State: Replica: fbac24feb0f74a2583264578c3dcf2b3, State: Running, Role: LEADER
I20260925 10:42:24.344203 5845 consensus_queue.cc:237] T ea42676e7b2e441087764f3874cfe8f5 P fbac24feb0f74a2583264578c3dcf2b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b41fee5f629349268798ff60f57a31bb" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 46473 } } peers { permanent_uuid: "899b0712f9d24f4a9a2e0d518962620d" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 34201 } } peers { permanent_uuid: "fbac24feb0f74a2583264578c3dcf2b3" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 36355 } }
I20260925 10:42:24.348678 5362 catalog_manager.cc:5495] T ea42676e7b2e441087764f3874cfe8f5 P fbac24feb0f74a2583264578c3dcf2b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to fbac24feb0f74a2583264578c3dcf2b3 (127.23.100.1). New cstate: current_term: 1 leader_uuid: "fbac24feb0f74a2583264578c3dcf2b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b41fee5f629349268798ff60f57a31bb" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 46473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "899b0712f9d24f4a9a2e0d518962620d" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 34201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbac24feb0f74a2583264578c3dcf2b3" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 36355 } health_report { overall_health: HEALTHY } } }
W20260925 10:42:24.372087 5820 tablet.cc:2369] T ea42676e7b2e441087764f3874cfe8f5 P b41fee5f629349268798ff60f57a31bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:42:24.383695 5687 tablet.cc:2369] T ea42676e7b2e441087764f3874cfe8f5 P 899b0712f9d24f4a9a2e0d518962620d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:42:24.389124 5835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:24.384307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45474 (local address 127.23.100.1:36355)
0925 10:42:24.384510 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:24.384514 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:24.384531 (+ 17us) server_negotiation.cc:407] Connection header received
0925 10:42:24.384739 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:24.384742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:24.384818 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:24.384937 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:42:24.385234 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:24.385908 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:24.387280 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:24.387672 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:24.387761 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:24.388194 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:42:24.388674 (+ 480us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:42:24.388777 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:24.388866 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 10:42:24.388929 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260925 10:42:24.397038 5553 tablet.cc:2369] T ea42676e7b2e441087764f3874cfe8f5 P fbac24feb0f74a2583264578c3dcf2b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:42:24.397045 5774 raft_consensus.cc:1261] T ea42676e7b2e441087764f3874cfe8f5 P b41fee5f629349268798ff60f57a31bb [term 1 FOLLOWER]: Refusing update from remote peer fbac24feb0f74a2583264578c3dcf2b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:42:24.397043 5641 raft_consensus.cc:1261] T ea42676e7b2e441087764f3874cfe8f5 P 899b0712f9d24f4a9a2e0d518962620d [term 1 FOLLOWER]: Refusing update from remote peer fbac24feb0f74a2583264578c3dcf2b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:42:24.398027 5848 consensus_queue.cc:1035] T ea42676e7b2e441087764f3874cfe8f5 P fbac24feb0f74a2583264578c3dcf2b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b41fee5f629349268798ff60f57a31bb" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 46473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:42:24.398402 5845 consensus_queue.cc:1035] T ea42676e7b2e441087764f3874cfe8f5 P fbac24feb0f74a2583264578c3dcf2b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "899b0712f9d24f4a9a2e0d518962620d" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 34201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:42:24.409279 5853 mvcc.cc:205] Tried to move back new op lower bound from 7333203740239982592 to 7333203740037500928. Current Snapshot: MvccSnapshot[applied={T|T < 7333203740239982592}]
I20260925 10:42:24.410387 5852 mvcc.cc:205] Tried to move back new op lower bound from 7333203740239982592 to 7333203740037500928. Current Snapshot: MvccSnapshot[applied={T|T < 7333203740239982592}]
I20260925 10:42:24.424772 5363 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52934:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:42:24.425030 5363 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52934:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:42:24.428429 5363 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611: Sending DeleteTablet for 3 replicas of tablet ea42676e7b2e441087764f3874cfe8f5
I20260925 10:42:24.429443 5754 tablet_service.cc:1514] Processing DeleteTablet for tablet ea42676e7b2e441087764f3874cfe8f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:42:24 UTC) from {username='kudu', principal='kudu/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:41226
I20260925 10:42:24.429610 5621 tablet_service.cc:1514] Processing DeleteTablet for tablet ea42676e7b2e441087764f3874cfe8f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:42:24 UTC) from {username='kudu', principal='kudu/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:41618
I20260925 10:42:24.429945 5487 tablet_service.cc:1514] Processing DeleteTablet for tablet ea42676e7b2e441087764f3874cfe8f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:42:24 UTC) from {username='kudu', principal='kudu/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:45468
I20260925 10:42:24.430990 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 5424
I20260925 10:42:24.440130 5863 tablet_replica.cc:330] T ea42676e7b2e441087764f3874cfe8f5 P fbac24feb0f74a2583264578c3dcf2b3: stopping tablet replica
I20260925 10:42:24.440671 5863 raft_consensus.cc:2229] T ea42676e7b2e441087764f3874cfe8f5 P fbac24feb0f74a2583264578c3dcf2b3 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:42:24.441645 5863 raft_consensus.cc:2258] T ea42676e7b2e441087764f3874cfe8f5 P fbac24feb0f74a2583264578c3dcf2b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:42:24.441833 5861 tablet_replica.cc:330] T ea42676e7b2e441087764f3874cfe8f5 P b41fee5f629349268798ff60f57a31bb: stopping tablet replica
I20260925 10:42:24.442324 5861 raft_consensus.cc:2229] T ea42676e7b2e441087764f3874cfe8f5 P b41fee5f629349268798ff60f57a31bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:42:24.442891 5861 raft_consensus.cc:2258] T ea42676e7b2e441087764f3874cfe8f5 P b41fee5f629349268798ff60f57a31bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:42:24.444545 5861 ts_tablet_manager.cc:1905] T ea42676e7b2e441087764f3874cfe8f5 P b41fee5f629349268798ff60f57a31bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:42:24.444566 5863 ts_tablet_manager.cc:1905] T ea42676e7b2e441087764f3874cfe8f5 P fbac24feb0f74a2583264578c3dcf2b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:42:24.449041 5861 ts_tablet_manager.cc:1918] T ea42676e7b2e441087764f3874cfe8f5 P b41fee5f629349268798ff60f57a31bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:42:24.449196 5861 log.cc:1196] T ea42676e7b2e441087764f3874cfe8f5 P b41fee5f629349268798ff60f57a31bb: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-2/wal/wals/ea42676e7b2e441087764f3874cfe8f5
I20260925 10:42:24.449739 5861 ts_tablet_manager.cc:1939] T ea42676e7b2e441087764f3874cfe8f5 P b41fee5f629349268798ff60f57a31bb: Deleting consensus metadata
I20260925 10:42:24.451628 5348 catalog_manager.cc:4841] TS b41fee5f629349268798ff60f57a31bb (127.23.100.3:46473): tablet ea42676e7b2e441087764f3874cfe8f5 (table test-table [id=4a646f4b415c46fc8a6118d0f37ddf0a]) successfully deleted
I20260925 10:42:24.451934 5862 tablet_replica.cc:330] T ea42676e7b2e441087764f3874cfe8f5 P 899b0712f9d24f4a9a2e0d518962620d: stopping tablet replica
I20260925 10:42:24.452348 5862 raft_consensus.cc:2229] T ea42676e7b2e441087764f3874cfe8f5 P 899b0712f9d24f4a9a2e0d518962620d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:42:24.452729 5863 ts_tablet_manager.cc:1918] T ea42676e7b2e441087764f3874cfe8f5 P fbac24feb0f74a2583264578c3dcf2b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:42:24.452934 5862 raft_consensus.cc:2258] T ea42676e7b2e441087764f3874cfe8f5 P 899b0712f9d24f4a9a2e0d518962620d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:42:24.452983 5863 log.cc:1196] T ea42676e7b2e441087764f3874cfe8f5 P fbac24feb0f74a2583264578c3dcf2b3: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-0/wal/wals/ea42676e7b2e441087764f3874cfe8f5
I20260925 10:42:24.453624 5863 ts_tablet_manager.cc:1939] T ea42676e7b2e441087764f3874cfe8f5 P fbac24feb0f74a2583264578c3dcf2b3: Deleting consensus metadata
I20260925 10:42:24.454978 5862 ts_tablet_manager.cc:1905] T ea42676e7b2e441087764f3874cfe8f5 P 899b0712f9d24f4a9a2e0d518962620d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:42:24.455327 5350 catalog_manager.cc:4841] TS fbac24feb0f74a2583264578c3dcf2b3 (127.23.100.1:36355): tablet ea42676e7b2e441087764f3874cfe8f5 (table test-table [id=4a646f4b415c46fc8a6118d0f37ddf0a]) successfully deleted
Sep 25 10:42:24 dist-test-slave-dxdd krb5kdc[5318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.1@KRBTEST.COM
I20260925 10:42:24.462265 5862 ts_tablet_manager.cc:1918] T ea42676e7b2e441087764f3874cfe8f5 P 899b0712f9d24f4a9a2e0d518962620d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:42:24.462442 5862 log.cc:1196] T ea42676e7b2e441087764f3874cfe8f5 P 899b0712f9d24f4a9a2e0d518962620d: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-1/wal/wals/ea42676e7b2e441087764f3874cfe8f5
I20260925 10:42:24.462961 5862 ts_tablet_manager.cc:1939] T ea42676e7b2e441087764f3874cfe8f5 P 899b0712f9d24f4a9a2e0d518962620d: Deleting consensus metadata
I20260925 10:42:24.464715 5347 catalog_manager.cc:4841] TS 899b0712f9d24f4a9a2e0d518962620d (127.23.100.2:34201): tablet ea42676e7b2e441087764f3874cfe8f5 (table test-table [id=4a646f4b415c46fc8a6118d0f37ddf0a]) successfully deleted
I20260925 10:42:24.466853 5835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:24.432267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45488 (local address 127.23.100.1:36355)
0925 10:42:24.432486 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:24.432495 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:24.449660 (+ 17165us) server_negotiation.cc:407] Connection header received
0925 10:42:24.450059 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:24.450075 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:24.450175 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:24.450264 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:42:24.452384 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:24.453707 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:24.455640 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:24.456112 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:24.456263 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:24.460611 (+ 4348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:42:24.460633 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:42:24.460645 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:42:24.460685 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:42:24.463869 (+ 3184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:24.464669 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:24.464679 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:24.464683 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:24.464851 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:24.465215 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:24.465221 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:24.465224 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:24.465670 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:42:24.465979 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:24.466187 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 10:42:24.466470 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 10:42:24.467581 5547 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:42:24.566032 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 5424
I20260925 10:42:24.585050 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 5558
Sep 25 10:42:24 dist-test-slave-dxdd krb5kdc[5318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.2@KRBTEST.COM
I20260925 10:42:24.599690 5836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:24.585887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41626 (local address 127.23.100.2:34201)
0925 10:42:24.586072 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:24.586078 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:24.586098 (+ 20us) server_negotiation.cc:407] Connection header received
0925 10:42:24.586245 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:24.586257 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:24.586360 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:24.586467 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:42:24.587687 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:24.588433 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:24.589814 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:24.590224 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:24.590404 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:24.594597 (+ 4193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:42:24.594615 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:42:24.594641 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:42:24.594688 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:42:24.597333 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:24.597943 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:24.597950 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:24.597952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:24.598073 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:24.598412 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:24.598415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:24.598417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:24.598803 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:42:24.599006 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:24.599267 (+ 261us) server_negotiation.cc:299] Negotiation successful
0925 10:42:24.599433 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 10:42:24.600368 5681 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:42:24.689558 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 5558
I20260925 10:42:24.706449 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 5691
Sep 25 10:42:24 dist-test-slave-dxdd krb5kdc[5318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.3@KRBTEST.COM
I20260925 10:42:24.721577 5834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:24.707216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41232 (local address 127.23.100.3:46473)
0925 10:42:24.707440 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:24.707446 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:24.707563 (+ 117us) server_negotiation.cc:407] Connection header received
0925 10:42:24.707772 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:24.707782 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:24.707854 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:24.707971 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:42:24.709375 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:24.710093 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:24.711586 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:24.711985 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:24.712123 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:24.716545 (+ 4422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:42:24.716559 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:42:24.716575 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:42:24.716610 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:42:24.719204 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:24.719838 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:24.719845 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:24.719847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:24.719967 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:24.720267 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:24.720270 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:24.720272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:24.720588 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:42:24.720795 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:24.721060 (+ 265us) server_negotiation.cc:299] Negotiation successful
0925 10:42:24.721239 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 10:42:24.722224 5814 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:42:24.810017 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 5691
I20260925 10:42:24.826522 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 5331
I20260925 10:42:24.827548 5393 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:42:24.937119 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 5331
I20260925 10:42:24.957942 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:44293
--webserver_interface=127.23.100.62
--webserver_port=34661
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:34751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.62:44293
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:42:25.278666 5869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:42:25.278976 5869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:42:25.279134 5869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:42:25.286134 5869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:42:25.286262 5869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:42:25.286389 5869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:42:25.286494 5869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:42:25.286571 5869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:42:25.296235 5869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:34751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.23.100.62:44293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:44293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.62
--webserver_port=34661
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:42:25.297479 5869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:42:25.298805 5869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:42:25.320739 5876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:42:25.320675 5875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:42:25.320703 5878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:42:25.321429 5869 server_base.cc:1034] running on GCE node
I20260925 10:42:25.322412 5869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:42:25.324779 5869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:42:25.326049 5869 hybrid_clock.cc:648] HybridClock initialized: now 1790332945325966 us; error 63 us; skew 500 ppm
I20260925 10:42:25.326867 5869 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:42:25 dist-test-slave-dxdd krb5kdc[5318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332945, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:42:25.332553 5869 init.cc:377] Logged in from keytab as kudu/127.23.100.62@KRBTEST.COM (short username kudu)
I20260925 10:42:25.335841 5869 webserver.cc:460] Webserver started at http://127.23.100.62:34661/ using document root <none> and password file <none>
I20260925 10:42:25.337004 5869 fs_manager.cc:359] Metadata directory not provided
I20260925 10:42:25.337210 5869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:42:25.345974 5869 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 10:42:25.349539 5885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:42:25.351444 5869 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260925 10:42:25.351643 5869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/master-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/master-0/wal
uuid: "120f316264344be088c2a7ab0c815611"
format_stamp: "Formatted at 2026-09-25 10:42:22 on dist-test-slave-dxdd"
server_key: "63bbc826e1c633d016bed2354f899237"
server_key_iv: "f0cf8bc7ae9cd7e2c8cec39a0d24542e"
server_key_version: "encryptionkey@0"
I20260925 10:42:25.353150 5869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:42:25.379066 5869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:42:25.383057 5869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:42:25.383327 5869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:42:25.401512 5869 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.62:44293
I20260925 10:42:25.401533 5936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.62:44293 every 8 connection(s)
I20260925 10:42:25.403072 5869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
I20260925 10:42:25.409807 5937 sys_catalog.cc:263] Verifying existing consensus state
I20260925 10:42:25.409989 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 5869
I20260925 10:42:25.410964 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.1:36355
--local_ip_for_outbound_sockets=127.23.100.1
--tserver_master_addrs=127.23.100.62:44293
--webserver_port=43109
--webserver_interface=127.23.100.1
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab
--principal=kudu/127.23.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:34751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:42:25.414846 5937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611: Bootstrap starting.
I20260925 10:42:25.435283 5937 log.cc:824] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611: Log is configured to *not* fsync() on all Append() calls
I20260925 10:42:25.446899 5937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:42:25.447413 5937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611: Bootstrap complete.
I20260925 10:42:25.456691 5937 raft_consensus.cc:348] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "120f316264344be088c2a7ab0c815611" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 44293 } }
I20260925 10:42:25.457505 5937 raft_consensus.cc:729] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 120f316264344be088c2a7ab0c815611, State: Initialized, Role: FOLLOWER
I20260925 10:42:25.458159 5937 consensus_queue.cc:260] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "120f316264344be088c2a7ab0c815611" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 44293 } }
I20260925 10:42:25.458468 5937 raft_consensus.cc:388] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:42:25.458609 5937 raft_consensus.cc:482] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:42:25.458811 5937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611 [term 1 FOLLOWER]: Advancing to term 2
I20260925 10:42:25.462571 5937 raft_consensus.cc:504] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "120f316264344be088c2a7ab0c815611" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 44293 } }
I20260925 10:42:25.463069 5937 leader_election.cc:302] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 120f316264344be088c2a7ab0c815611; no voters:
I20260925 10:42:25.463704 5937 leader_election.cc:288] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 10:42:25.463832 5941 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 10:42:25.465083 5941 raft_consensus.cc:686] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611 [term 2 LEADER]: Becoming Leader. State: Replica: 120f316264344be088c2a7ab0c815611, State: Running, Role: LEADER
I20260925 10:42:25.465643 5941 consensus_queue.cc:237] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "120f316264344be088c2a7ab0c815611" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 44293 } }
I20260925 10:42:25.465951 5937 sys_catalog.cc:564] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:42:25.468044 5943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 120f316264344be088c2a7ab0c815611. Latest consensus state: current_term: 2 leader_uuid: "120f316264344be088c2a7ab0c815611" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "120f316264344be088c2a7ab0c815611" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 44293 } } }
I20260925 10:42:25.468444 5943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611 [sys.catalog]: This master's current role is: LEADER
I20260925 10:42:25.469012 5942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "120f316264344be088c2a7ab0c815611" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "120f316264344be088c2a7ab0c815611" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 44293 } } }
I20260925 10:42:25.469332 5942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611 [sys.catalog]: This master's current role is: LEADER
I20260925 10:42:25.472563 5947 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:42:25.481940 5947 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:42:25.483739 5947 catalog_manager.cc:1259] Loaded cluster ID: e67f1f3cc2224ce1903cb5b534b8f813
I20260925 10:42:25.483906 5947 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:42:25.487777 5947 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 10:42:25.512897 5947 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 10:42:25.524674 5947 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:42:25.526695 5947 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611: Loaded TSK: 0
I20260925 10:42:25.528491 5947 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 10:42:25.751456 5939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:42:25.751698 5939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:42:25.751844 5939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:42:25.759189 5939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:42:25.759289 5939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:42:25.759459 5939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.1
I20260925 10:42:25.769033 5939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:34751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab
--principal=kudu/127.23.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.1:36355
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.1
--webserver_port=43109
--tserver_master_addrs=127.23.100.62:44293
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.1
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:42:25.770460 5939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:42:25.771715 5939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:42:25.786057 5964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:42:25.786067 5965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:42:25.786976 5967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:42:25.787348 5939 server_base.cc:1034] running on GCE node
I20260925 10:42:25.788026 5939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:42:25.789034 5939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:42:25.790267 5939 hybrid_clock.cc:648] HybridClock initialized: now 1790332945790215 us; error 47 us; skew 500 ppm
I20260925 10:42:25.790594 5939 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:42:25 dist-test-slave-dxdd krb5kdc[5318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332945, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:42:25.798288 5939 init.cc:377] Logged in from keytab as kudu/127.23.100.1@KRBTEST.COM (short username kudu)
I20260925 10:42:25.799818 5939 webserver.cc:460] Webserver started at http://127.23.100.1:43109/ using document root <none> and password file <none>
I20260925 10:42:25.800470 5939 fs_manager.cc:359] Metadata directory not provided
I20260925 10:42:25.800637 5939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:42:25.805773 5939 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 10:42:25.808559 5974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:42:25.809944 5939 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:42:25.810117 5939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-0/wal
uuid: "fbac24feb0f74a2583264578c3dcf2b3"
format_stamp: "Formatted at 2026-09-25 10:42:23 on dist-test-slave-dxdd"
server_key: "1e916f473bc49b2cacaf9185d2fc9817"
server_key_iv: "f91717781fb3e3deee175992a8ce98dc"
server_key_version: "encryptionkey@0"
I20260925 10:42:25.811122 5939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:42:25.829377 5939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:42:25.833375 5939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:42:25.833652 5939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:42:25.834740 5939 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:42:25.836102 5939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:42:25.836254 5939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:42:25.836388 5939 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:42:25.836504 5939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:42:25.869065 5939 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.1:36355
I20260925 10:42:25.869148 6086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.1:36355 every 8 connection(s)
I20260925 10:42:25.870587 5939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
I20260925 10:42:25.876876 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 5939
I20260925 10:42:25.877463 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.2:34201
--local_ip_for_outbound_sockets=127.23.100.2
--tserver_master_addrs=127.23.100.62:44293
--webserver_port=42147
--webserver_interface=127.23.100.2
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab
--principal=kudu/127.23.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:34751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 10:42:25 dist-test-slave-dxdd krb5kdc[5318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332945, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.1@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:42:25.896996 6090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:25.873638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.1:59413)
0925 10:42:25.875510 (+ 1872us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:25.875553 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 10:42:25.877523 (+ 1970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:42:25.879181 (+ 1658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:42:25.879188 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:42:25.879678 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:42:25.880204 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:25.880222 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:25.882670 (+ 2448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:25.882674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:25.883861 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:25.883873 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:25.885388 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:25.885398 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:25.885729 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:25.886546 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:42:25.886580 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:42:25.889303 (+ 2723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:42:25.892730 (+ 3427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:42:25.892739 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:42:25.892754 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:42:25.893178 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:42:25.893623 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:42:25.893628 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:42:25.893631 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:42:25.893810 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:42:25.894583 (+ 773us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:42:25.894590 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:42:25.894901 (+ 311us) client_negotiation.cc:769] Sending connection context
0925 10:42:25.895158 (+ 257us) client_negotiation.cc:240] Negotiation successful
0925 10:42:25.895688 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1590,"thread_start_us":1356,"threads_started":1}
I20260925 10:42:25.896996 6089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:25.873338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:59413 (local address 127.23.100.62:44293)
0925 10:42:25.874731 (+ 1393us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:25.874743 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:25.875699 (+ 956us) server_negotiation.cc:407] Connection header received
0925 10:42:25.878139 (+ 2440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:25.878161 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:25.878649 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:25.879082 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:42:25.880446 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:25.882516 (+ 2070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:25.884012 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:25.885127 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:25.885272 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:25.889599 (+ 4327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:42:25.889608 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:42:25.889622 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:42:25.889658 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:42:25.892584 (+ 2926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:25.893325 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:25.893334 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:25.893346 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:25.893502 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:25.893922 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:25.893927 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:25.893930 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:25.894440 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:42:25.894630 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:25.895463 (+ 833us) server_negotiation.cc:299] Negotiation successful
0925 10:42:25.895839 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1190,"thread_start_us":1045,"threads_started":1}
I20260925 10:42:25.900319 6087 heartbeater.cc:347] Connected to a master server at 127.23.100.62:44293
I20260925 10:42:25.900616 6087 heartbeater.cc:464] Registering TS with master...
I20260925 10:42:25.901328 6087 heartbeater.cc:511] Master 127.23.100.62:44293 requested a full tablet report, sending...
I20260925 10:42:25.903681 5902 ts_manager.cc:194] Registered new tserver with Master: fbac24feb0f74a2583264578c3dcf2b3 (127.23.100.1:36355)
I20260925 10:42:25.905819 5902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.1@KRBTEST.COM'} at 127.23.100.1:59413
W20260925 10:42:26.206143 6091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:42:26.206504 6091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:42:26.206662 6091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:42:26.213917 6091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:42:26.214069 6091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:42:26.214228 6091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.2
I20260925 10:42:26.223846 6091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:34751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab
--principal=kudu/127.23.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.2:34201
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.2
--webserver_port=42147
--tserver_master_addrs=127.23.100.62:44293
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.2
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:42:26.225026 6091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:42:26.226380 6091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:42:26.239192 6097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:42:26.239214 6096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:42:26.239151 6099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:42:26.240177 6091 server_base.cc:1034] running on GCE node
I20260925 10:42:26.240798 6091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:42:26.241786 6091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:42:26.243014 6091 hybrid_clock.cc:648] HybridClock initialized: now 1790332946242955 us; error 49 us; skew 500 ppm
I20260925 10:42:26.243319 6091 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:42:26 dist-test-slave-dxdd krb5kdc[5318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332946, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:42:26.246506 6091 init.cc:377] Logged in from keytab as kudu/127.23.100.2@KRBTEST.COM (short username kudu)
I20260925 10:42:26.247936 6091 webserver.cc:460] Webserver started at http://127.23.100.2:42147/ using document root <none> and password file <none>
I20260925 10:42:26.248594 6091 fs_manager.cc:359] Metadata directory not provided
I20260925 10:42:26.248754 6091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:42:26.253999 6091 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:42:26.256589 6106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:42:26.257920 6091 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 10:42:26.258126 6091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-1/wal
uuid: "899b0712f9d24f4a9a2e0d518962620d"
format_stamp: "Formatted at 2026-09-25 10:42:23 on dist-test-slave-dxdd"
server_key: "befdc78e3b4c2dcb2ed0d22bd73f1efa"
server_key_iv: "2533722dd23d7c65d40a21e0e1c9e344"
server_key_version: "encryptionkey@0"
I20260925 10:42:26.259133 6091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:42:26.273116 6091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:42:26.277267 6091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:42:26.277536 6091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:42:26.278472 6091 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:42:26.279723 6091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:42:26.279839 6091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:42:26.279958 6091 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:42:26.280040 6091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:42:26.312505 6091 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.2:34201
I20260925 10:42:26.312558 6218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.2:34201 every 8 connection(s)
I20260925 10:42:26.313902 6091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
I20260925 10:42:26.322716 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 6091
I20260925 10:42:26.323308 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.3:46473
--local_ip_for_outbound_sockets=127.23.100.3
--tserver_master_addrs=127.23.100.62:44293
--webserver_port=37913
--webserver_interface=127.23.100.3
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab
--principal=kudu/127.23.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:34751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 10:42:26 dist-test-slave-dxdd krb5kdc[5318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332946, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.2@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:42:26.333676 6089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:26.316468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:57419 (local address 127.23.100.62:44293)
0925 10:42:26.316639 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:26.316643 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:26.317703 (+ 1060us) server_negotiation.cc:407] Connection header received
0925 10:42:26.319517 (+ 1814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:26.319523 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:26.319620 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:26.319727 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:42:26.321138 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:26.321835 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:26.323570 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:26.323974 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:26.324045 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:26.327546 (+ 3501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:42:26.327551 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:42:26.327554 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:42:26.327616 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:42:26.330580 (+ 2964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:26.331266 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:26.331274 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:26.331280 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:26.331438 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:26.331812 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:26.331820 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:26.331826 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:26.332192 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:42:26.332338 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:26.332966 (+ 628us) server_negotiation.cc:299] Negotiation successful
0925 10:42:26.333229 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 10:42:26.334290 6221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:26.316856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.2:57419)
0925 10:42:26.317552 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:26.317595 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 10:42:26.319240 (+ 1645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:42:26.319956 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:42:26.319963 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:42:26.320440 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:42:26.320971 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:26.320986 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:26.321995 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:26.321999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:26.323357 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:26.323390 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:26.324292 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:26.324297 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:26.324480 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:26.325177 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:42:26.325198 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:42:26.327335 (+ 2137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:42:26.330719 (+ 3384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:42:26.330725 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:42:26.330737 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:42:26.331143 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:42:26.331537 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:42:26.331540 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:42:26.331542 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:42:26.331703 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:42:26.332302 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:42:26.332307 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:42:26.332609 (+ 302us) client_negotiation.cc:769] Sending connection context
0925 10:42:26.332819 (+ 210us) client_negotiation.cc:240] Negotiation successful
0925 10:42:26.333188 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":281,"threads_started":1}
I20260925 10:42:26.335727 6219 heartbeater.cc:347] Connected to a master server at 127.23.100.62:44293
I20260925 10:42:26.336032 6219 heartbeater.cc:464] Registering TS with master...
I20260925 10:42:26.336740 6219 heartbeater.cc:511] Master 127.23.100.62:44293 requested a full tablet report, sending...
I20260925 10:42:26.338167 5902 ts_manager.cc:194] Registered new tserver with Master: 899b0712f9d24f4a9a2e0d518962620d (127.23.100.2:34201)
I20260925 10:42:26.338997 5902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.2@KRBTEST.COM'} at 127.23.100.2:57419
W20260925 10:42:26.639889 6222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:42:26.640139 6222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:42:26.640304 6222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:42:26.647388 6222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:42:26.647501 6222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:42:26.647679 6222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.3
I20260925 10:42:26.657095 6222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:34751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab
--principal=kudu/127.23.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.3:46473
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.3
--webserver_port=37913
--tserver_master_addrs=127.23.100.62:44293
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.3
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:42:26.658337 6222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:42:26.659722 6222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:42:26.671887 6227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:42:26.671851 6228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:42:26.671921 6230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:42:26.672672 6222 server_base.cc:1034] running on GCE node
I20260925 10:42:26.673270 6222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:42:26.674109 6222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:42:26.675343 6222 hybrid_clock.cc:648] HybridClock initialized: now 1790332946675278 us; error 68 us; skew 500 ppm
I20260925 10:42:26.675654 6222 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:42:26 dist-test-slave-dxdd krb5kdc[5318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332946, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:42:26.678515 6222 init.cc:377] Logged in from keytab as kudu/127.23.100.3@KRBTEST.COM (short username kudu)
I20260925 10:42:26.679883 6222 webserver.cc:460] Webserver started at http://127.23.100.3:37913/ using document root <none> and password file <none>
I20260925 10:42:26.680503 6222 fs_manager.cc:359] Metadata directory not provided
I20260925 10:42:26.680660 6222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:42:26.685498 6222 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:42:26.688107 6237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:42:26.689431 6222 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 10:42:26.689549 6222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-2/wal
uuid: "b41fee5f629349268798ff60f57a31bb"
format_stamp: "Formatted at 2026-09-25 10:42:24 on dist-test-slave-dxdd"
server_key: "95b3064ccfbe0b6d74a36075a4017263"
server_key_iv: "216d497b9da00f220a1e90fbbf32cc30"
server_key_version: "encryptionkey@0"
I20260925 10:42:26.690482 6222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:42:26.718669 6222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:42:26.722539 6222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:42:26.722770 6222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:42:26.723670 6222 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:42:26.724920 6222 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:42:26.724999 6222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:42:26.725073 6222 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:42:26.725129 6222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:42:26.757474 6222 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.3:46473
I20260925 10:42:26.757593 6349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.3:46473 every 8 connection(s)
I20260925 10:42:26.759107 6222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
I20260925 10:42:26.767048 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 6222
Sep 25 10:42:26 dist-test-slave-dxdd krb5kdc[5318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332946, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.3@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:42:26.779989 6089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:26.761447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:54295 (local address 127.23.100.62:44293)
0925 10:42:26.761656 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:26.761662 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:26.762897 (+ 1235us) server_negotiation.cc:407] Connection header received
0925 10:42:26.764744 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:26.764748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:26.764812 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:26.764898 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:42:26.766396 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:26.767321 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:26.769501 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:26.769931 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:26.770079 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:26.774024 (+ 3945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:42:26.774031 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:42:26.774035 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:42:26.774075 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:42:26.776817 (+ 2742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:26.777698 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:26.777705 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:26.777711 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:26.777848 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:26.778247 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:26.778253 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:26.778256 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:26.778579 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:42:26.778737 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:26.779402 (+ 665us) server_negotiation.cc:299] Negotiation successful
0925 10:42:26.779654 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 10:42:26.781085 6352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:26.761933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.3:54295)
0925 10:42:26.762705 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:26.762765 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 10:42:26.764498 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:42:26.765087 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:42:26.765095 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:42:26.765579 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:42:26.766207 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:26.766224 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:26.767482 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:26.767496 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:26.769346 (+ 1850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:26.769358 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:26.770157 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:26.770166 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:26.770474 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:26.771238 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:42:26.771268 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:42:26.773838 (+ 2570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:42:26.776978 (+ 3140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:42:26.776995 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:42:26.777011 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:42:26.777590 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:42:26.777954 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:42:26.777957 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:42:26.777959 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:42:26.778132 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:42:26.778723 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:42:26.778730 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:42:26.779035 (+ 305us) client_negotiation.cc:769] Sending connection context
0925 10:42:26.779292 (+ 257us) client_negotiation.cc:240] Negotiation successful
0925 10:42:26.779799 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":322,"threads_started":1}
I20260925 10:42:26.782516 6350 heartbeater.cc:347] Connected to a master server at 127.23.100.62:44293
I20260925 10:42:26.782822 6350 heartbeater.cc:464] Registering TS with master...
I20260925 10:42:26.783071 6354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:26.768066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52946 (local address 127.23.100.62:44293)
0925 10:42:26.769088 (+ 1022us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:26.769092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:26.769106 (+ 14us) server_negotiation.cc:407] Connection header received
0925 10:42:26.769143 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:26.769146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:26.769205 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:26.769320 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:42:26.770996 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:26.772171 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:26.773789 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:26.774503 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:26.774630 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:26.777553 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:42:26.777559 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:42:26.777563 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:42:26.777589 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:42:26.780446 (+ 2857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:26.781100 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:26.781106 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:26.781111 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:26.781281 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:26.781644 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:26.781654 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:26.781657 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:26.781907 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:42:26.782181 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:26.782483 (+ 302us) server_negotiation.cc:299] Negotiation successful
0925 10:42:26.782705 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":924,"thread_start_us":806,"threads_started":1}
I20260925 10:42:26.783736 6350 heartbeater.cc:511] Master 127.23.100.62:44293 requested a full tablet report, sending...
I20260925 10:42:26.785346 5901 ts_manager.cc:194] Registered new tserver with Master: b41fee5f629349268798ff60f57a31bb (127.23.100.3:46473)
I20260925 10:42:26.786190 5901 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.3@KRBTEST.COM'} at 127.23.100.3:54295
I20260925 10:42:26.798758 23952 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:42:26.804801 6354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:26.799571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52948 (local address 127.23.100.62:44293)
0925 10:42:26.799750 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:26.799757 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:26.800147 (+ 390us) server_negotiation.cc:407] Connection header received
0925 10:42:26.800297 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:26.800300 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:26.800419 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:26.800528 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:42:26.800867 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:26.801528 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:26.802965 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:26.803358 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:26.803447 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:26.803904 (+ 457us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:42:26.804321 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:42:26.804417 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:26.804503 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 10:42:26.804554 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 10:42:26.805570 5901 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:52948:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:42:26.812532 5901 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:42:26.843971 6361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:26.823479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41632 (local address 127.23.100.2:34201)
0925 10:42:26.834628 (+ 11149us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:26.834638 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:26.835522 (+ 884us) server_negotiation.cc:407] Connection header received
0925 10:42:26.835642 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:26.835650 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:26.835913 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:26.836160 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:42:26.836949 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:26.838533 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:26.841454 (+ 2921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:26.842839 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:26.842956 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:26.843103 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:26.843477 (+ 374us) server_negotiation.cc:299] Negotiation successful
0925 10:42:26.843649 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10952,"thread_start_us":10797,"threads_started":1}
I20260925 10:42:26.844077 6362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:26.823415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:36355 (local address 127.0.0.1:45502)
0925 10:42:26.836396 (+ 12981us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:26.836410 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:42:26.836560 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:42:26.837119 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:42:26.837125 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:42:26.837152 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:42:26.837509 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:26.837520 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:26.839430 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:26.839437 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:26.841113 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:26.841124 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:26.843248 (+ 2124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:26.843253 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:26.843380 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:26.843437 (+ 57us) client_negotiation.cc:769] Sending connection context
0925 10:42:26.843520 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 10:42:26.843655 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12824,"mutex_wait_us":37,"thread_start_us":12445,"threads_started":1}
I20260925 10:42:26.844133 6360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:26.823163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.2:34201 (local address 127.0.0.1:41632)
0925 10:42:26.835061 (+ 11898us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:26.835109 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 10:42:26.835326 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:42:26.836197 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:42:26.836210 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:42:26.836241 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:42:26.836778 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:26.836790 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:26.838670 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:26.838675 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:26.841302 (+ 2627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:26.841317 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:26.843031 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:26.843037 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:26.843234 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:26.843284 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 10:42:26.843422 (+ 138us) client_negotiation.cc:240] Negotiation successful
0925 10:42:26.843650 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11623,"mutex_wait_us":60,"thread_start_us":11472,"threads_started":1}
I20260925 10:42:26.844960 6364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:26.825623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.3:46473 (local address 127.0.0.1:41248)
0925 10:42:26.836486 (+ 10863us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:26.836504 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:42:26.836624 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:42:26.837350 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:42:26.837355 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:42:26.837381 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:42:26.837779 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:26.837790 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:26.839691 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:26.839697 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:26.841710 (+ 2013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:26.841720 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:26.844247 (+ 2527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:26.844251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:26.844432 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:26.844469 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 10:42:26.844560 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 10:42:26.844680 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10695,"thread_start_us":9484,"threads_started":1}
I20260925 10:42:26.845108 6363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:26.823603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45502 (local address 127.23.100.1:36355)
0925 10:42:26.835533 (+ 11930us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:26.835541 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:26.836512 (+ 971us) server_negotiation.cc:407] Connection header received
0925 10:42:26.836721 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:26.836727 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:26.836922 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:26.837117 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:42:26.837676 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:26.839249 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:26.841261 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:26.842743 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:26.842893 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:26.843099 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:26.844724 (+ 1625us) server_negotiation.cc:299] Negotiation successful
0925 10:42:26.844862 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11753,"thread_start_us":11950,"threads_started":1}
I20260925 10:42:26.845690 6365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:26.826396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41248 (local address 127.23.100.3:46473)
0925 10:42:26.832899 (+ 6503us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:26.832912 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:26.836758 (+ 3846us) server_negotiation.cc:407] Connection header received
0925 10:42:26.836862 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:26.836872 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:26.837100 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:26.837279 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:42:26.837972 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:26.839545 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:26.842121 (+ 2576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:26.843630 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:26.843769 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:26.844003 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:26.845197 (+ 1194us) server_negotiation.cc:299] Negotiation successful
0925 10:42:26.845357 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6305,"thread_start_us":6141,"threads_started":1}
I20260925 10:42:26.848047 6154 tablet_service.cc:1467] Processing CreateTablet for tablet 63f331376be845c9b84765336e7d6553 (DEFAULT_TABLE table=test-table [id=ef24c34a47804d50a2face7dfed38e4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:42:26.848519 6022 tablet_service.cc:1467] Processing CreateTablet for tablet 63f331376be845c9b84765336e7d6553 (DEFAULT_TABLE table=test-table [id=ef24c34a47804d50a2face7dfed38e4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:42:26.849221 6154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63f331376be845c9b84765336e7d6553. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:42:26.848994 6285 tablet_service.cc:1467] Processing CreateTablet for tablet 63f331376be845c9b84765336e7d6553 (DEFAULT_TABLE table=test-table [id=ef24c34a47804d50a2face7dfed38e4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:42:26.850008 6022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63f331376be845c9b84765336e7d6553. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:42:26.850236 6285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63f331376be845c9b84765336e7d6553. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:42:26.861194 6366 tablet_bootstrap.cc:492] T 63f331376be845c9b84765336e7d6553 P 899b0712f9d24f4a9a2e0d518962620d: Bootstrap starting.
I20260925 10:42:26.865280 6367 tablet_bootstrap.cc:492] T 63f331376be845c9b84765336e7d6553 P fbac24feb0f74a2583264578c3dcf2b3: Bootstrap starting.
I20260925 10:42:26.865638 6366 tablet_bootstrap.cc:654] T 63f331376be845c9b84765336e7d6553 P 899b0712f9d24f4a9a2e0d518962620d: Neither blocks nor log segments found. Creating new log.
I20260925 10:42:26.867185 6368 tablet_bootstrap.cc:492] T 63f331376be845c9b84765336e7d6553 P b41fee5f629349268798ff60f57a31bb: Bootstrap starting.
I20260925 10:42:26.868124 6366 log.cc:824] T 63f331376be845c9b84765336e7d6553 P 899b0712f9d24f4a9a2e0d518962620d: Log is configured to *not* fsync() on all Append() calls
I20260925 10:42:26.869266 6367 tablet_bootstrap.cc:654] T 63f331376be845c9b84765336e7d6553 P fbac24feb0f74a2583264578c3dcf2b3: Neither blocks nor log segments found. Creating new log.
I20260925 10:42:26.870710 6368 tablet_bootstrap.cc:654] T 63f331376be845c9b84765336e7d6553 P b41fee5f629349268798ff60f57a31bb: Neither blocks nor log segments found. Creating new log.
I20260925 10:42:26.870990 6367 log.cc:824] T 63f331376be845c9b84765336e7d6553 P fbac24feb0f74a2583264578c3dcf2b3: Log is configured to *not* fsync() on all Append() calls
I20260925 10:42:26.872143 6368 log.cc:824] T 63f331376be845c9b84765336e7d6553 P b41fee5f629349268798ff60f57a31bb: Log is configured to *not* fsync() on all Append() calls
I20260925 10:42:26.878772 6366 tablet_bootstrap.cc:492] T 63f331376be845c9b84765336e7d6553 P 899b0712f9d24f4a9a2e0d518962620d: No bootstrap required, opened a new log
I20260925 10:42:26.879099 6366 ts_tablet_manager.cc:1397] T 63f331376be845c9b84765336e7d6553 P 899b0712f9d24f4a9a2e0d518962620d: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.008s
I20260925 10:42:26.884331 6368 tablet_bootstrap.cc:492] T 63f331376be845c9b84765336e7d6553 P b41fee5f629349268798ff60f57a31bb: No bootstrap required, opened a new log
I20260925 10:42:26.884605 6368 ts_tablet_manager.cc:1397] T 63f331376be845c9b84765336e7d6553 P b41fee5f629349268798ff60f57a31bb: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.006s
I20260925 10:42:26.886760 6367 tablet_bootstrap.cc:492] T 63f331376be845c9b84765336e7d6553 P fbac24feb0f74a2583264578c3dcf2b3: No bootstrap required, opened a new log
I20260925 10:42:26.887022 6367 ts_tablet_manager.cc:1397] T 63f331376be845c9b84765336e7d6553 P fbac24feb0f74a2583264578c3dcf2b3: Time spent bootstrapping tablet: real 0.022s user 0.016s sys 0.004s
I20260925 10:42:26.892443 6368 raft_consensus.cc:348] T 63f331376be845c9b84765336e7d6553 P b41fee5f629349268798ff60f57a31bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "899b0712f9d24f4a9a2e0d518962620d" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 34201 } } peers { permanent_uuid: "fbac24feb0f74a2583264578c3dcf2b3" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 36355 } } peers { permanent_uuid: "b41fee5f629349268798ff60f57a31bb" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 46473 } }
I20260925 10:42:26.892834 6368 raft_consensus.cc:374] T 63f331376be845c9b84765336e7d6553 P b41fee5f629349268798ff60f57a31bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:42:26.892927 6368 raft_consensus.cc:729] T 63f331376be845c9b84765336e7d6553 P b41fee5f629349268798ff60f57a31bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b41fee5f629349268798ff60f57a31bb, State: Initialized, Role: FOLLOWER
I20260925 10:42:26.893251 6366 raft_consensus.cc:348] T 63f331376be845c9b84765336e7d6553 P 899b0712f9d24f4a9a2e0d518962620d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "899b0712f9d24f4a9a2e0d518962620d" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 34201 } } peers { permanent_uuid: "fbac24feb0f74a2583264578c3dcf2b3" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 36355 } } peers { permanent_uuid: "b41fee5f629349268798ff60f57a31bb" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 46473 } }
I20260925 10:42:26.893718 6366 raft_consensus.cc:374] T 63f331376be845c9b84765336e7d6553 P 899b0712f9d24f4a9a2e0d518962620d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:42:26.893823 6366 raft_consensus.cc:729] T 63f331376be845c9b84765336e7d6553 P 899b0712f9d24f4a9a2e0d518962620d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 899b0712f9d24f4a9a2e0d518962620d, State: Initialized, Role: FOLLOWER
I20260925 10:42:26.893685 6368 consensus_queue.cc:260] T 63f331376be845c9b84765336e7d6553 P b41fee5f629349268798ff60f57a31bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "899b0712f9d24f4a9a2e0d518962620d" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 34201 } } peers { permanent_uuid: "fbac24feb0f74a2583264578c3dcf2b3" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 36355 } } peers { permanent_uuid: "b41fee5f629349268798ff60f57a31bb" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 46473 } }
I20260925 10:42:26.894418 6366 consensus_queue.cc:260] T 63f331376be845c9b84765336e7d6553 P 899b0712f9d24f4a9a2e0d518962620d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "899b0712f9d24f4a9a2e0d518962620d" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 34201 } } peers { permanent_uuid: "fbac24feb0f74a2583264578c3dcf2b3" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 36355 } } peers { permanent_uuid: "b41fee5f629349268798ff60f57a31bb" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 46473 } }
I20260925 10:42:26.895031 6350 heartbeater.cc:503] Master 127.23.100.62:44293 was elected leader, sending a full tablet report...
I20260925 10:42:26.895654 6368 ts_tablet_manager.cc:1428] T 63f331376be845c9b84765336e7d6553 P b41fee5f629349268798ff60f57a31bb: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260925 10:42:26.896090 6219 heartbeater.cc:503] Master 127.23.100.62:44293 was elected leader, sending a full tablet report...
I20260925 10:42:26.896178 6366 ts_tablet_manager.cc:1428] T 63f331376be845c9b84765336e7d6553 P 899b0712f9d24f4a9a2e0d518962620d: Time spent starting tablet: real 0.017s user 0.014s sys 0.000s
I20260925 10:42:26.899820 6367 raft_consensus.cc:348] T 63f331376be845c9b84765336e7d6553 P fbac24feb0f74a2583264578c3dcf2b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "899b0712f9d24f4a9a2e0d518962620d" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 34201 } } peers { permanent_uuid: "fbac24feb0f74a2583264578c3dcf2b3" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 36355 } } peers { permanent_uuid: "b41fee5f629349268798ff60f57a31bb" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 46473 } }
I20260925 10:42:26.900251 6367 raft_consensus.cc:374] T 63f331376be845c9b84765336e7d6553 P fbac24feb0f74a2583264578c3dcf2b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:42:26.900362 6367 raft_consensus.cc:729] T 63f331376be845c9b84765336e7d6553 P fbac24feb0f74a2583264578c3dcf2b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbac24feb0f74a2583264578c3dcf2b3, State: Initialized, Role: FOLLOWER
I20260925 10:42:26.900969 6367 consensus_queue.cc:260] T 63f331376be845c9b84765336e7d6553 P fbac24feb0f74a2583264578c3dcf2b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "899b0712f9d24f4a9a2e0d518962620d" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 34201 } } peers { permanent_uuid: "fbac24feb0f74a2583264578c3dcf2b3" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 36355 } } peers { permanent_uuid: "b41fee5f629349268798ff60f57a31bb" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 46473 } }
I20260925 10:42:26.902068 6087 heartbeater.cc:503] Master 127.23.100.62:44293 was elected leader, sending a full tablet report...
I20260925 10:42:26.902607 6367 ts_tablet_manager.cc:1428] T 63f331376be845c9b84765336e7d6553 P fbac24feb0f74a2583264578c3dcf2b3: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260925 10:42:26.909973 6372 raft_consensus.cc:482] T 63f331376be845c9b84765336e7d6553 P b41fee5f629349268798ff60f57a31bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:42:26.910171 6372 raft_consensus.cc:504] T 63f331376be845c9b84765336e7d6553 P b41fee5f629349268798ff60f57a31bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "899b0712f9d24f4a9a2e0d518962620d" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 34201 } } peers { permanent_uuid: "fbac24feb0f74a2583264578c3dcf2b3" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 36355 } } peers { permanent_uuid: "b41fee5f629349268798ff60f57a31bb" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 46473 } }
I20260925 10:42:26.911576 6372 leader_election.cc:288] T 63f331376be845c9b84765336e7d6553 P b41fee5f629349268798ff60f57a31bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 899b0712f9d24f4a9a2e0d518962620d (127.23.100.2:34201), fbac24feb0f74a2583264578c3dcf2b3 (127.23.100.1:36355)
I20260925 10:42:26.917660 6361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:26.912077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:44393 (local address 127.23.100.2:34201)
0925 10:42:26.912446 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:26.912453 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:26.912472 (+ 19us) server_negotiation.cc:407] Connection header received
0925 10:42:26.912534 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:26.912539 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:26.912624 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:26.912718 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:42:26.913280 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:26.914284 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:26.916046 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:26.916803 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:26.916975 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:26.917047 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:26.917287 (+ 240us) server_negotiation.cc:299] Negotiation successful
0925 10:42:26.917413 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238}
I20260925 10:42:26.917678 6352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:26.911816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.2:34201 (local address 127.23.100.3:44393)
0925 10:42:26.912203 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:26.912225 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 10:42:26.912357 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:42:26.912879 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:42:26.912884 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:42:26.912911 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:42:26.913125 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:26.913134 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:26.914460 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:26.914466 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:26.915890 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:26.915909 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:26.916956 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:26.916962 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:26.917135 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:26.917159 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 10:42:26.917263 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 10:42:26.917387 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":26}
I20260925 10:42:26.918444 6363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:26.912103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:40779 (local address 127.23.100.1:36355)
0925 10:42:26.912274 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:26.912279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:26.913049 (+ 770us) server_negotiation.cc:407] Connection header received
0925 10:42:26.913320 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:26.913324 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:26.913382 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:26.913464 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:42:26.913985 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:26.914752 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:26.916500 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:26.917433 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:26.917585 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:26.917655 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:26.918062 (+ 407us) server_negotiation.cc:299] Negotiation successful
0925 10:42:26.918178 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 10:42:26.918532 6375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:26.912076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:36355 (local address 127.23.100.3:40779)
0925 10:42:26.912956 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:26.912978 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 10:42:26.913133 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:42:26.913507 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:42:26.913512 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:42:26.913538 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:42:26.913841 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:26.913852 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:26.914897 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:26.914902 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:26.916355 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:26.916367 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:26.917642 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:26.917648 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:26.917878 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:26.917907 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 10:42:26.918027 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 10:42:26.918144 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":738,"mutex_wait_us":2,"thread_start_us":568,"threads_started":1}
I20260925 10:42:26.918597 6174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f331376be845c9b84765336e7d6553" candidate_uuid: "b41fee5f629349268798ff60f57a31bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "899b0712f9d24f4a9a2e0d518962620d" is_pre_election: true
I20260925 10:42:26.919083 6174 raft_consensus.cc:2454] T 63f331376be845c9b84765336e7d6553 P 899b0712f9d24f4a9a2e0d518962620d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b41fee5f629349268798ff60f57a31bb in term 0.
I20260925 10:42:26.919427 6042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f331376be845c9b84765336e7d6553" candidate_uuid: "b41fee5f629349268798ff60f57a31bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbac24feb0f74a2583264578c3dcf2b3" is_pre_election: true
I20260925 10:42:26.919713 6238 leader_election.cc:302] T 63f331376be845c9b84765336e7d6553 P b41fee5f629349268798ff60f57a31bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 899b0712f9d24f4a9a2e0d518962620d, b41fee5f629349268798ff60f57a31bb; no voters:
I20260925 10:42:26.919826 6042 raft_consensus.cc:2454] T 63f331376be845c9b84765336e7d6553 P fbac24feb0f74a2583264578c3dcf2b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b41fee5f629349268798ff60f57a31bb in term 0.
I20260925 10:42:26.920094 6372 raft_consensus.cc:2781] T 63f331376be845c9b84765336e7d6553 P b41fee5f629349268798ff60f57a31bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:42:26.920240 6372 raft_consensus.cc:482] T 63f331376be845c9b84765336e7d6553 P b41fee5f629349268798ff60f57a31bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:42:26.920349 6372 raft_consensus.cc:3037] T 63f331376be845c9b84765336e7d6553 P b41fee5f629349268798ff60f57a31bb [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:42:26.922537 6372 raft_consensus.cc:504] T 63f331376be845c9b84765336e7d6553 P b41fee5f629349268798ff60f57a31bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "899b0712f9d24f4a9a2e0d518962620d" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 34201 } } peers { permanent_uuid: "fbac24feb0f74a2583264578c3dcf2b3" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 36355 } } peers { permanent_uuid: "b41fee5f629349268798ff60f57a31bb" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 46473 } }
I20260925 10:42:26.923059 6372 leader_election.cc:288] T 63f331376be845c9b84765336e7d6553 P b41fee5f629349268798ff60f57a31bb [CANDIDATE]: Term 1 election: Requested vote from peers 899b0712f9d24f4a9a2e0d518962620d (127.23.100.2:34201), fbac24feb0f74a2583264578c3dcf2b3 (127.23.100.1:36355)
I20260925 10:42:26.923412 6174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f331376be845c9b84765336e7d6553" candidate_uuid: "b41fee5f629349268798ff60f57a31bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "899b0712f9d24f4a9a2e0d518962620d"
I20260925 10:42:26.923485 6042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f331376be845c9b84765336e7d6553" candidate_uuid: "b41fee5f629349268798ff60f57a31bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbac24feb0f74a2583264578c3dcf2b3"
I20260925 10:42:26.923625 6174 raft_consensus.cc:3037] T 63f331376be845c9b84765336e7d6553 P 899b0712f9d24f4a9a2e0d518962620d [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:42:26.923688 6042 raft_consensus.cc:3037] T 63f331376be845c9b84765336e7d6553 P fbac24feb0f74a2583264578c3dcf2b3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:42:26.925721 6042 raft_consensus.cc:2454] T 63f331376be845c9b84765336e7d6553 P fbac24feb0f74a2583264578c3dcf2b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b41fee5f629349268798ff60f57a31bb in term 1.
I20260925 10:42:26.925752 6174 raft_consensus.cc:2454] T 63f331376be845c9b84765336e7d6553 P 899b0712f9d24f4a9a2e0d518962620d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b41fee5f629349268798ff60f57a31bb in term 1.
I20260925 10:42:26.926084 6241 leader_election.cc:302] T 63f331376be845c9b84765336e7d6553 P b41fee5f629349268798ff60f57a31bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b41fee5f629349268798ff60f57a31bb, fbac24feb0f74a2583264578c3dcf2b3; no voters:
I20260925 10:42:26.926379 6372 raft_consensus.cc:2781] T 63f331376be845c9b84765336e7d6553 P b41fee5f629349268798ff60f57a31bb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:42:26.926785 6372 raft_consensus.cc:686] T 63f331376be845c9b84765336e7d6553 P b41fee5f629349268798ff60f57a31bb [term 1 LEADER]: Becoming Leader. State: Replica: b41fee5f629349268798ff60f57a31bb, State: Running, Role: LEADER
I20260925 10:42:26.927217 6372 consensus_queue.cc:237] T 63f331376be845c9b84765336e7d6553 P b41fee5f629349268798ff60f57a31bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "899b0712f9d24f4a9a2e0d518962620d" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 34201 } } peers { permanent_uuid: "fbac24feb0f74a2583264578c3dcf2b3" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 36355 } } peers { permanent_uuid: "b41fee5f629349268798ff60f57a31bb" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 46473 } }
I20260925 10:42:26.931555 5902 catalog_manager.cc:5495] T 63f331376be845c9b84765336e7d6553 P b41fee5f629349268798ff60f57a31bb reported cstate change: term changed from 0 to 1, leader changed from <none> to b41fee5f629349268798ff60f57a31bb (127.23.100.3). New cstate: current_term: 1 leader_uuid: "b41fee5f629349268798ff60f57a31bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "899b0712f9d24f4a9a2e0d518962620d" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 34201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbac24feb0f74a2583264578c3dcf2b3" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 36355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b41fee5f629349268798ff60f57a31bb" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 46473 } health_report { overall_health: HEALTHY } } }
I20260925 10:42:26.968246 6365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:26.963266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41256 (local address 127.23.100.3:46473)
0925 10:42:26.963469 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:26.963473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:26.963500 (+ 27us) server_negotiation.cc:407] Connection header received
0925 10:42:26.963652 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:26.963656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:26.963740 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:26.963860 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:42:26.964155 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:26.964852 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:26.966247 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:26.966697 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:26.966788 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:26.967221 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:42:26.967694 (+ 473us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:42:26.967797 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:26.967890 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 10:42:26.967949 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 10:42:26.975889 6042 raft_consensus.cc:1261] T 63f331376be845c9b84765336e7d6553 P fbac24feb0f74a2583264578c3dcf2b3 [term 1 FOLLOWER]: Refusing update from remote peer b41fee5f629349268798ff60f57a31bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:42:26.975893 6174 raft_consensus.cc:1261] T 63f331376be845c9b84765336e7d6553 P 899b0712f9d24f4a9a2e0d518962620d [term 1 FOLLOWER]: Refusing update from remote peer b41fee5f629349268798ff60f57a31bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:42:26.976814 6376 consensus_queue.cc:1035] T 63f331376be845c9b84765336e7d6553 P b41fee5f629349268798ff60f57a31bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "899b0712f9d24f4a9a2e0d518962620d" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 34201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:42:26.977137 6372 consensus_queue.cc:1035] T 63f331376be845c9b84765336e7d6553 P b41fee5f629349268798ff60f57a31bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbac24feb0f74a2583264578c3dcf2b3" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 36355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:42:26.987897 6380 mvcc.cc:205] Tried to move back new op lower bound from 7333203750803984384 to 7333203750617718784. Current Snapshot: MvccSnapshot[applied={T|T < 7333203750803984384}]
I20260925 10:42:26.988052 6379 mvcc.cc:205] Tried to move back new op lower bound from 7333203750803984384 to 7333203750617718784. Current Snapshot: MvccSnapshot[applied={T|T < 7333203750803984384}]
I20260925 10:42:26.995388 6381 mvcc.cc:205] Tried to move back new op lower bound from 7333203750803984384 to 7333203750617718784. Current Snapshot: MvccSnapshot[applied={T|T < 7333203750803984384}]
I20260925 10:42:27.004313 5901 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:52948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:42:27.004598 5901 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:52948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:42:27.008049 5901 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611: Sending DeleteTablet for 3 replicas of tablet 63f331376be845c9b84765336e7d6553
I20260925 10:42:27.009133 6154 tablet_service.cc:1514] Processing DeleteTablet for tablet 63f331376be845c9b84765336e7d6553 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:42:27 UTC) from {username='kudu', principal='kudu/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:41632
I20260925 10:42:27.009536 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 5939
I20260925 10:42:27.009511 6022 tablet_service.cc:1514] Processing DeleteTablet for tablet 63f331376be845c9b84765336e7d6553 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:42:27 UTC) from {username='kudu', principal='kudu/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:45502
I20260925 10:42:27.011021 6285 tablet_service.cc:1514] Processing DeleteTablet for tablet 63f331376be845c9b84765336e7d6553 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:42:27 UTC) from {username='kudu', principal='kudu/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:41248
I20260925 10:42:27.033329 6363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:27.010477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45510 (local address 127.23.100.1:36355)
0925 10:42:27.010732 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:27.010740 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:27.010759 (+ 19us) server_negotiation.cc:407] Connection header received
0925 10:42:27.011011 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:27.011020 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:27.011122 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:27.011246 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:42:27.013265 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:27.014968 (+ 1703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:27.017261 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:27.018155 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:27.018367 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:27.022733 (+ 4366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:42:27.022754 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:42:27.022778 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:42:27.022845 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:42:27.027912 (+ 5067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:27.029095 (+ 1183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:27.029108 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:27.029114 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:27.029304 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:27.029854 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:27.029862 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:27.029868 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:27.030475 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:42:27.030850 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:27.032480 (+ 1630us) server_negotiation.cc:299] Negotiation successful
0925 10:42:27.032802 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260925 10:42:27.034456 6082 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:42:27.034996 6391 tablet_replica.cc:330] T 63f331376be845c9b84765336e7d6553 P b41fee5f629349268798ff60f57a31bb: stopping tablet replica
I20260925 10:42:27.035656 6391 raft_consensus.cc:2229] T 63f331376be845c9b84765336e7d6553 P b41fee5f629349268798ff60f57a31bb [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:42:27.036688 6391 raft_consensus.cc:2258] T 63f331376be845c9b84765336e7d6553 P b41fee5f629349268798ff60f57a31bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:42:27.037521 6389 tablet_replica.cc:330] T 63f331376be845c9b84765336e7d6553 P 899b0712f9d24f4a9a2e0d518962620d: stopping tablet replica
I20260925 10:42:27.037930 6389 raft_consensus.cc:2229] T 63f331376be845c9b84765336e7d6553 P 899b0712f9d24f4a9a2e0d518962620d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:42:27.038504 6389 raft_consensus.cc:2258] T 63f331376be845c9b84765336e7d6553 P 899b0712f9d24f4a9a2e0d518962620d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:42:27.039251 6391 ts_tablet_manager.cc:1905] T 63f331376be845c9b84765336e7d6553 P b41fee5f629349268798ff60f57a31bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:42:27.041311 6389 ts_tablet_manager.cc:1905] T 63f331376be845c9b84765336e7d6553 P 899b0712f9d24f4a9a2e0d518962620d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:42:27.047370 6391 ts_tablet_manager.cc:1918] T 63f331376be845c9b84765336e7d6553 P b41fee5f629349268798ff60f57a31bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:42:27.047516 6391 log.cc:1196] T 63f331376be845c9b84765336e7d6553 P b41fee5f629349268798ff60f57a31bb: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-2/wal/wals/63f331376be845c9b84765336e7d6553
I20260925 10:42:27.048077 6391 ts_tablet_manager.cc:1939] T 63f331376be845c9b84765336e7d6553 P b41fee5f629349268798ff60f57a31bb: Deleting consensus metadata
I20260925 10:42:27.048285 6389 ts_tablet_manager.cc:1918] T 63f331376be845c9b84765336e7d6553 P 899b0712f9d24f4a9a2e0d518962620d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:42:27.048450 6389 log.cc:1196] T 63f331376be845c9b84765336e7d6553 P 899b0712f9d24f4a9a2e0d518962620d: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-1/wal/wals/63f331376be845c9b84765336e7d6553
I20260925 10:42:27.049002 6389 ts_tablet_manager.cc:1939] T 63f331376be845c9b84765336e7d6553 P 899b0712f9d24f4a9a2e0d518962620d: Deleting consensus metadata
I20260925 10:42:27.049842 5887 catalog_manager.cc:4841] TS b41fee5f629349268798ff60f57a31bb (127.23.100.3:46473): tablet 63f331376be845c9b84765336e7d6553 (table test-table [id=ef24c34a47804d50a2face7dfed38e4a]) successfully deleted
I20260925 10:42:27.050720 5886 catalog_manager.cc:4841] TS 899b0712f9d24f4a9a2e0d518962620d (127.23.100.2:34201): tablet 63f331376be845c9b84765336e7d6553 (table test-table [id=ef24c34a47804d50a2face7dfed38e4a]) successfully deleted
I20260925 10:42:27.172947 6390 tablet_replica.cc:330] T 63f331376be845c9b84765336e7d6553 P fbac24feb0f74a2583264578c3dcf2b3: stopping tablet replica
I20260925 10:42:27.173179 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 5939
I20260925 10:42:27.190945 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 6091
W20260925 10:42:27.190968 5889 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.100.1:36355 (error 108)
I20260925 10:42:27.192919 6364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:27.192468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:36355 (local address 127.0.0.1:45512)
0925 10:42:27.192655 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:27.192766 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.1:36355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260925 10:42:27.193231 5889 catalog_manager.cc:4584] TS fbac24feb0f74a2583264578c3dcf2b3 (127.23.100.1:36355): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 63f331376be845c9b84765336e7d6553: Network error: Client connection negotiation failed: client connection to 127.23.100.1:36355: connect: Connection refused (error 111)
I20260925 10:42:27.206095 6361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:27.192174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41642 (local address 127.23.100.2:34201)
0925 10:42:27.192351 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:27.192355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:27.192371 (+ 16us) server_negotiation.cc:407] Connection header received
0925 10:42:27.192423 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:27.192427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:27.192482 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:27.192605 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:42:27.194393 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:27.195164 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:27.197005 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:27.197385 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:27.197473 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:27.200749 (+ 3276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:42:27.200761 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:42:27.200775 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:42:27.200818 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:42:27.203567 (+ 2749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:27.204364 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:27.204370 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:27.204372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:27.204483 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:27.204852 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:27.204855 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:27.204857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:27.205178 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:42:27.205387 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:27.205709 (+ 322us) server_negotiation.cc:299] Negotiation successful
0925 10:42:27.205850 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 10:42:27.206826 6214 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:42:27.246927 6364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:27.246386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:36355 (local address 127.0.0.1:45514)
0925 10:42:27.246612 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:27.246729 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.1:36355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260925 10:42:27.248065 6364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:27.247675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:36355 (local address 127.0.0.1:45522)
0925 10:42:27.247833 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:27.247914 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.1:36355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 10:42:27.294220 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 6091
I20260925 10:42:27.308588 6364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:27.308154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:36355 (local address 127.0.0.1:45526)
0925 10:42:27.308347 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:27.308429 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.1:36355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 10:42:27.309623 6364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:27.309307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:36355 (local address 127.0.0.1:45542)
0925 10:42:27.309440 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:27.309514 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.1:36355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 10:42:27.316659 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 6222
I20260925 10:42:27.336812 6365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:27.321695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41266 (local address 127.23.100.3:46473)
0925 10:42:27.321864 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:27.321869 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:27.321935 (+ 66us) server_negotiation.cc:407] Connection header received
0925 10:42:27.322120 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:27.322127 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:27.322189 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:27.322275 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:42:27.324217 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:27.325000 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:27.326940 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:27.327391 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:27.327481 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:27.330951 (+ 3470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:42:27.330964 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:42:27.330977 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:42:27.331011 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:42:27.333912 (+ 2901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:27.334819 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:27.334827 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:27.334829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:27.334982 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:27.335398 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:27.335401 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:27.335403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:27.335735 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:42:27.335973 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:27.336307 (+ 334us) server_negotiation.cc:299] Negotiation successful
0925 10:42:27.336455 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 10:42:27.337468 6345 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:42:27.390110 6364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:27.389684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:36355 (local address 127.0.0.1:45556)
0925 10:42:27.389877 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:27.389961 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.1:36355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260925 10:42:27.391089 6364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:27.390741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:36355 (local address 127.0.0.1:45560)
0925 10:42:27.390928 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:27.390989 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.1:36355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260925 10:42:27.456635 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 6222
I20260925 10:42:27.476250 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 5869
I20260925 10:42:27.477349 5932 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:42:27.588376 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 5869
I20260925 10:42:27.588506 6364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:27.587966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:36355 (local address 127.0.0.1:45576)
0925 10:42:27.588164 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:27.588261 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.1:36355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 10:42:27.609537 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:44293
--webserver_interface=127.23.100.62
--webserver_port=34661
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:34751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.62:44293
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:42:27.931026 6397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:42:27.931277 6397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:42:27.931450 6397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:42:27.938287 6397 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:42:27.938463 6397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:42:27.938560 6397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:42:27.938653 6397 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:42:27.938731 6397 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:42:27.948118 6397 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:34751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.62:44293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:44293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.62
--webserver_port=34661
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:42:27.949352 6397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:42:27.950702 6397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:42:27.962702 6402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:42:27.962611 6403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:42:27.962671 6405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:42:27.963577 6397 server_base.cc:1034] running on GCE node
I20260925 10:42:27.964242 6397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:42:27.965540 6397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:42:27.966740 6397 hybrid_clock.cc:648] HybridClock initialized: now 1790332947966679 us; error 49 us; skew 500 ppm
I20260925 10:42:27.967029 6397 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:42:27 dist-test-slave-dxdd krb5kdc[5318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332947, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:42:27.969969 6397 init.cc:377] Logged in from keytab as kudu/127.23.100.62@KRBTEST.COM (short username kudu)
I20260925 10:42:27.971383 6397 webserver.cc:460] Webserver started at http://127.23.100.62:34661/ using document root <none> and password file <none>
I20260925 10:42:27.971994 6397 fs_manager.cc:359] Metadata directory not provided
I20260925 10:42:27.972095 6397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:42:27.977188 6397 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260925 10:42:27.979748 6412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:42:27.980983 6397 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 10:42:27.981109 6397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/master-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/master-0/wal
uuid: "120f316264344be088c2a7ab0c815611"
format_stamp: "Formatted at 2026-09-25 10:42:22 on dist-test-slave-dxdd"
server_key: "63bbc826e1c633d016bed2354f899237"
server_key_iv: "f0cf8bc7ae9cd7e2c8cec39a0d24542e"
server_key_version: "encryptionkey@0"
I20260925 10:42:27.981967 6397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:42:27.996222 6397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:42:27.999869 6397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:42:28.000078 6397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:42:28.017884 6397 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.62:44293
I20260925 10:42:28.017902 6463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.62:44293 every 8 connection(s)
I20260925 10:42:28.019308 6397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
I20260925 10:42:28.025830 6464 sys_catalog.cc:263] Verifying existing consensus state
I20260925 10:42:28.028970 6464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611: Bootstrap starting.
I20260925 10:42:28.029835 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 6397
I20260925 10:42:28.030839 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.1:36355
--local_ip_for_outbound_sockets=127.23.100.1
--tserver_master_addrs=127.23.100.62:44293
--webserver_port=43109
--webserver_interface=127.23.100.1
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab
--principal=kudu/127.23.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:34751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:42:28.052169 6464 log.cc:824] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611: Log is configured to *not* fsync() on all Append() calls
I20260925 10:42:28.068253 6464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:42:28.068854 6464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611: Bootstrap complete.
I20260925 10:42:28.077842 6464 raft_consensus.cc:348] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "120f316264344be088c2a7ab0c815611" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 44293 } }
I20260925 10:42:28.078780 6464 raft_consensus.cc:729] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 120f316264344be088c2a7ab0c815611, State: Initialized, Role: FOLLOWER
I20260925 10:42:28.079384 6464 consensus_queue.cc:260] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "120f316264344be088c2a7ab0c815611" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 44293 } }
I20260925 10:42:28.079622 6464 raft_consensus.cc:388] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:42:28.079738 6464 raft_consensus.cc:482] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:42:28.079918 6464 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611 [term 2 FOLLOWER]: Advancing to term 3
I20260925 10:42:28.083581 6464 raft_consensus.cc:504] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "120f316264344be088c2a7ab0c815611" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 44293 } }
I20260925 10:42:28.084049 6464 leader_election.cc:302] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 120f316264344be088c2a7ab0c815611; no voters:
I20260925 10:42:28.084739 6464 leader_election.cc:288] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 10:42:28.084854 6468 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 10:42:28.086254 6468 raft_consensus.cc:686] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611 [term 3 LEADER]: Becoming Leader. State: Replica: 120f316264344be088c2a7ab0c815611, State: Running, Role: LEADER
I20260925 10:42:28.086827 6468 consensus_queue.cc:237] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "120f316264344be088c2a7ab0c815611" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 44293 } }
I20260925 10:42:28.087025 6464 sys_catalog.cc:564] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:42:28.089097 6470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 120f316264344be088c2a7ab0c815611. Latest consensus state: current_term: 3 leader_uuid: "120f316264344be088c2a7ab0c815611" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "120f316264344be088c2a7ab0c815611" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 44293 } } }
I20260925 10:42:28.089428 6470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611 [sys.catalog]: This master's current role is: LEADER
I20260925 10:42:28.090003 6469 sys_catalog.cc:455] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "120f316264344be088c2a7ab0c815611" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "120f316264344be088c2a7ab0c815611" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 44293 } } }
I20260925 10:42:28.090270 6469 sys_catalog.cc:458] T 00000000000000000000000000000000 P 120f316264344be088c2a7ab0c815611 [sys.catalog]: This master's current role is: LEADER
I20260925 10:42:28.099737 6474 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:42:28.108331 6474 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:42:28.109710 6474 catalog_manager.cc:1259] Loaded cluster ID: e67f1f3cc2224ce1903cb5b534b8f813
I20260925 10:42:28.109870 6474 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 10:42:28.111990 6474 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fb0d14c29ad google::LogMessage::Fail() at ??:0
@ 0x7fb0d14c588a google::LogMessage::SendToLog() at ??:0
@ 0x7fb0d14c24de google::LogMessage::Flush() at ??:0
@ 0x7fb0d14c3a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb0e6ef5512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fb0e6ef2976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
I20260925 10:42:28.134799 6486 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7fb0e6f7a610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fb0e6f7a439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb0d410e345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb0d411191c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fb0d4111799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb0d40eceaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb0d73c66db start_thread at ??:0
@ 0x7fb0d035c61f clone at ??:0
W20260925 10:42:28.389670 6466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:42:28.389976 6466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:42:28.390142 6466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:42:28.397392 6466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:42:28.397528 6466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:42:28.397706 6466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.1
I20260925 10:42:28.407217 6466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:34751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab
--principal=kudu/127.23.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.1:36355
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.1
--webserver_port=43109
--tserver_master_addrs=127.23.100.62:44293
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.1
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:42:28.408438 6466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:42:28.409694 6466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:42:28.422493 6506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:42:28.422573 6508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:42:28.422550 6505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:42:28.423318 6466 server_base.cc:1034] running on GCE node
I20260925 10:42:28.423902 6466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:42:28.424777 6466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:42:28.425995 6466 hybrid_clock.cc:648] HybridClock initialized: now 1790332948425946 us; error 44 us; skew 500 ppm
I20260925 10:42:28.426343 6466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:42:28 dist-test-slave-dxdd krb5kdc[5318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332948, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:42:28.429382 6466 init.cc:377] Logged in from keytab as kudu/127.23.100.1@KRBTEST.COM (short username kudu)
I20260925 10:42:28.430722 6466 webserver.cc:460] Webserver started at http://127.23.100.1:43109/ using document root <none> and password file <none>
I20260925 10:42:28.431358 6466 fs_manager.cc:359] Metadata directory not provided
I20260925 10:42:28.431509 6466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:42:28.436547 6466 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20260925 10:42:28.439191 6515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:42:28.440441 6466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 10:42:28.440608 6466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-0/wal
uuid: "fbac24feb0f74a2583264578c3dcf2b3"
format_stamp: "Formatted at 2026-09-25 10:42:23 on dist-test-slave-dxdd"
server_key: "1e916f473bc49b2cacaf9185d2fc9817"
server_key_iv: "f91717781fb3e3deee175992a8ce98dc"
server_key_version: "encryptionkey@0"
I20260925 10:42:28.441499 6466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:42:28.459758 6466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:42:28.463337 6466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:42:28.463604 6466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:42:28.464500 6466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:42:28.466084 6522 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260925 10:42:28.471274 6466 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 10:42:28.471369 6466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260925 10:42:28.471524 6466 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 10:42:28.474602 6466 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 10:42:28.474675 6466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.003s sys 0.000s
I20260925 10:42:28.474979 6522 tablet_bootstrap.cc:492] T 63f331376be845c9b84765336e7d6553 P fbac24feb0f74a2583264578c3dcf2b3: Bootstrap starting.
I20260925 10:42:28.502236 6522 log.cc:824] T 63f331376be845c9b84765336e7d6553 P fbac24feb0f74a2583264578c3dcf2b3: Log is configured to *not* fsync() on all Append() calls
I20260925 10:42:28.512794 6466 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.1:36355
I20260925 10:42:28.512845 6629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.1:36355 every 8 connection(s)
I20260925 10:42:28.514003 6466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
I20260925 10:42:28.515661 6522 tablet_bootstrap.cc:492] T 63f331376be845c9b84765336e7d6553 P fbac24feb0f74a2583264578c3dcf2b3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:42:28.516245 6522 tablet_bootstrap.cc:492] T 63f331376be845c9b84765336e7d6553 P fbac24feb0f74a2583264578c3dcf2b3: Bootstrap complete.
I20260925 10:42:28.516906 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 6466
I20260925 10:42:28.517632 6522 ts_tablet_manager.cc:1397] T 63f331376be845c9b84765336e7d6553 P fbac24feb0f74a2583264578c3dcf2b3: Time spent bootstrapping tablet: real 0.043s user 0.032s sys 0.008s
I20260925 10:42:28.517729 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.2:34201
--local_ip_for_outbound_sockets=127.23.100.2
--tserver_master_addrs=127.23.100.62:44293
--webserver_port=42147
--webserver_interface=127.23.100.2
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab
--principal=kudu/127.23.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:34751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:42:28.519605 6632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:28.516964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.1:45337)
0925 10:42:28.518136 (+ 1172us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:28.518471 (+ 335us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":367,"threads_started":1}
W20260925 10:42:28.520310 6630 heartbeater.cc:644] Failed to heartbeat to 127.23.100.62:44293 (0 consecutive failures): Network error: Failed to ping master at 127.23.100.62:44293: Client connection negotiation failed: client connection to 127.23.100.62:44293: connect: Connection refused (error 111)
I20260925 10:42:28.521332 6632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:28.520857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.1:58385)
0925 10:42:28.521065 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:28.521166 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 10:42:28.522462 6632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:28.521985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.1:46929)
0925 10:42:28.522164 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:28.522269 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 10:42:28.522778 6630 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 10:42:28.526081 6522 raft_consensus.cc:348] T 63f331376be845c9b84765336e7d6553 P fbac24feb0f74a2583264578c3dcf2b3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "899b0712f9d24f4a9a2e0d518962620d" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 34201 } } peers { permanent_uuid: "fbac24feb0f74a2583264578c3dcf2b3" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 36355 } } peers { permanent_uuid: "b41fee5f629349268798ff60f57a31bb" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 46473 } }
I20260925 10:42:28.527509 6522 raft_consensus.cc:729] T 63f331376be845c9b84765336e7d6553 P fbac24feb0f74a2583264578c3dcf2b3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbac24feb0f74a2583264578c3dcf2b3, State: Initialized, Role: FOLLOWER
I20260925 10:42:28.528151 6522 consensus_queue.cc:260] T 63f331376be845c9b84765336e7d6553 P fbac24feb0f74a2583264578c3dcf2b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "899b0712f9d24f4a9a2e0d518962620d" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 34201 } } peers { permanent_uuid: "fbac24feb0f74a2583264578c3dcf2b3" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 36355 } } peers { permanent_uuid: "b41fee5f629349268798ff60f57a31bb" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 46473 } }
I20260925 10:42:28.529604 6632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:28.529297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.1:45089)
0925 10:42:28.529431 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:28.529500 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 10:42:28.529786 6522 ts_tablet_manager.cc:1428] T 63f331376be845c9b84765336e7d6553 P fbac24feb0f74a2583264578c3dcf2b3: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260925 10:42:28.530619 6632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:28.530292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.1:42977)
0925 10:42:28.530435 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:28.530518 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260925 10:42:28.838644 6633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:42:28.838922 6633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:42:28.839088 6633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:42:28.846431 6633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:42:28.846558 6633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:42:28.846773 6633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.2
I20260925 10:42:28.856664 6633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:34751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab
--principal=kudu/127.23.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.2:34201
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.2
--webserver_port=42147
--tserver_master_addrs=127.23.100.62:44293
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.2
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:42:28.857976 6633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:42:28.859287 6633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:42:28.871894 6639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:42:28.872063 6642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:42:28.871857 6640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:42:28.872709 6633 server_base.cc:1034] running on GCE node
I20260925 10:42:28.873302 6633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:42:28.874152 6633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:42:28.875375 6633 hybrid_clock.cc:648] HybridClock initialized: now 1790332948875334 us; error 60 us; skew 500 ppm
I20260925 10:42:28.875692 6633 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:42:28 dist-test-slave-dxdd krb5kdc[5318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332948, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:42:28.878719 6633 init.cc:377] Logged in from keytab as kudu/127.23.100.2@KRBTEST.COM (short username kudu)
I20260925 10:42:28.880012 6633 webserver.cc:460] Webserver started at http://127.23.100.2:42147/ using document root <none> and password file <none>
I20260925 10:42:28.880653 6633 fs_manager.cc:359] Metadata directory not provided
I20260925 10:42:28.880779 6633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:42:28.886799 6633 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 10:42:28.891086 6649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:42:28.892395 6633 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:42:28.892534 6633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-1/wal
uuid: "899b0712f9d24f4a9a2e0d518962620d"
format_stamp: "Formatted at 2026-09-25 10:42:23 on dist-test-slave-dxdd"
server_key: "befdc78e3b4c2dcb2ed0d22bd73f1efa"
server_key_iv: "2533722dd23d7c65d40a21e0e1c9e344"
server_key_version: "encryptionkey@0"
I20260925 10:42:28.893410 6633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:42:28.920800 6633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:42:28.924471 6633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:42:28.924685 6633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:42:28.925550 6633 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:42:28.926791 6633 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:42:28.926872 6633 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:42:28.926961 6633 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:42:28.927016 6633 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:42:28.960583 6633 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.2:34201
I20260925 10:42:28.960668 6761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.2:34201 every 8 connection(s)
I20260925 10:42:28.961990 6633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
I20260925 10:42:28.963074 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 6633
I20260925 10:42:28.963845 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.3:46473
--local_ip_for_outbound_sockets=127.23.100.3
--tserver_master_addrs=127.23.100.62:44293
--webserver_port=37913
--webserver_interface=127.23.100.3
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab
--principal=kudu/127.23.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:34751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:42:28.967516 6764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:28.965285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.2:47269)
0925 10:42:28.966101 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:28.966396 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":304,"threads_started":1}
W20260925 10:42:28.968235 6762 heartbeater.cc:644] Failed to heartbeat to 127.23.100.62:44293 (0 consecutive failures): Network error: Failed to ping master at 127.23.100.62:44293: Client connection negotiation failed: client connection to 127.23.100.62:44293: connect: Connection refused (error 111)
I20260925 10:42:28.969146 6764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:28.968733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.2:34071)
0925 10:42:28.968917 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:28.969007 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 10:42:28.970067 6764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:28.969661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.2:48883)
0925 10:42:28.969825 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:28.969918 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 10:42:28.970337 6762 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 10:42:29.284072 6765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:42:29.284336 6765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:42:29.284474 6765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:42:29.291356 6765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:42:29.291487 6765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:42:29.291684 6765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.3
I20260925 10:42:29.301054 6765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:34751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab
--principal=kudu/127.23.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.3:46473
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.3
--webserver_port=37913
--tserver_master_addrs=127.23.100.62:44293
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.3
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:42:29.302206 6765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:42:29.303486 6765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:42:29.315900 6771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:42:29.315989 6770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:42:29.315864 6773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:42:29.316532 6765 server_base.cc:1034] running on GCE node
I20260925 10:42:29.317099 6765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:42:29.317952 6765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:42:29.319168 6765 hybrid_clock.cc:648] HybridClock initialized: now 1790332949319129 us; error 61 us; skew 500 ppm
I20260925 10:42:29.319525 6765 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:42:29 dist-test-slave-dxdd krb5kdc[5318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332949, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:42:29.322569 6765 init.cc:377] Logged in from keytab as kudu/127.23.100.3@KRBTEST.COM (short username kudu)
I20260925 10:42:29.323864 6765 webserver.cc:460] Webserver started at http://127.23.100.3:37913/ using document root <none> and password file <none>
I20260925 10:42:29.324527 6765 fs_manager.cc:359] Metadata directory not provided
I20260925 10:42:29.324683 6765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:42:29.329524 6765 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:42:29.332011 6780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:42:29.333316 6765 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260925 10:42:29.333534 6765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-2/wal
uuid: "b41fee5f629349268798ff60f57a31bb"
format_stamp: "Formatted at 2026-09-25 10:42:24 on dist-test-slave-dxdd"
server_key: "95b3064ccfbe0b6d74a36075a4017263"
server_key_iv: "216d497b9da00f220a1e90fbbf32cc30"
server_key_version: "encryptionkey@0"
I20260925 10:42:29.334550 6765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:42:29.347703 6765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:42:29.351243 6765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:42:29.351533 6765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:42:29.352499 6765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:42:29.353833 6765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:42:29.353979 6765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:42:29.354113 6765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:42:29.354215 6765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:42:29.387030 6765 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.3:46473
I20260925 10:42:29.387115 6892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.3:46473 every 8 connection(s)
I20260925 10:42:29.388459 6765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
I20260925 10:42:29.392997 6895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:29.391340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.3:50269)
0925 10:42:29.392005 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:29.392197 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":257,"threads_started":1}
W20260925 10:42:29.393692 6893 heartbeater.cc:644] Failed to heartbeat to 127.23.100.62:44293 (0 consecutive failures): Network error: Failed to ping master at 127.23.100.62:44293: Client connection negotiation failed: client connection to 127.23.100.62:44293: connect: Connection refused (error 111)
I20260925 10:42:29.394479 6895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:29.394108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.3:55231)
0925 10:42:29.394249 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:29.394360 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 10:42:29.395298 6895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:29.395016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.3:55235)
0925 10:42:29.395134 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:29.395201 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 10:42:29.395515 6893 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 10:42:29.397113 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 6765
I20260925 10:42:29.532001 6897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:29.531325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.1:50251)
0925 10:42:29.531722 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:29.531822 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":203,"threads_started":1}
I20260925 10:42:29.684437 6898 raft_consensus.cc:482] T 63f331376be845c9b84765336e7d6553 P fbac24feb0f74a2583264578c3dcf2b3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:42:29.684723 6898 raft_consensus.cc:504] T 63f331376be845c9b84765336e7d6553 P fbac24feb0f74a2583264578c3dcf2b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "899b0712f9d24f4a9a2e0d518962620d" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 34201 } } peers { permanent_uuid: "fbac24feb0f74a2583264578c3dcf2b3" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 36355 } } peers { permanent_uuid: "b41fee5f629349268798ff60f57a31bb" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 46473 } }
I20260925 10:42:29.686388 6898 leader_election.cc:288] T 63f331376be845c9b84765336e7d6553 P fbac24feb0f74a2583264578c3dcf2b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 899b0712f9d24f4a9a2e0d518962620d (127.23.100.2:34201), b41fee5f629349268798ff60f57a31bb (127.23.100.3:46473)
Sep 25 10:42:29 dist-test-slave-dxdd krb5kdc[5318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332948, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.1@KRBTEST.COM for kudu/127.23.100.3@KRBTEST.COM
Sep 25 10:42:29 dist-test-slave-dxdd krb5kdc[5318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332948, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.1@KRBTEST.COM for kudu/127.23.100.2@KRBTEST.COM
I20260925 10:42:29.706291 6899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:29.686806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.3:46473 (local address 127.23.100.1:46855)
0925 10:42:29.687867 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:29.687889 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 10:42:29.688014 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:42:29.690638 (+ 2624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:42:29.690649 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:42:29.691355 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:42:29.692013 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:29.692030 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:29.693830 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:29.693835 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:29.695267 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:29.695279 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:29.696277 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:29.696284 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:29.696556 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:29.697328 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:42:29.697379 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:42:29.699710 (+ 2331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:42:29.702843 (+ 3133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:42:29.702851 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:42:29.702863 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:42:29.703250 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:42:29.703833 (+ 583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:42:29.703836 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:42:29.703839 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:42:29.704113 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:42:29.704847 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:42:29.704854 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:42:29.705158 (+ 304us) client_negotiation.cc:769] Sending connection context
0925 10:42:29.705400 (+ 242us) client_negotiation.cc:240] Negotiation successful
0925 10:42:29.705833 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":913,"mutex_wait_us":139,"spinlock_wait_cycles":56576,"thread_start_us":765,"threads_started":1}
I20260925 10:42:29.706642 6897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:29.686514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.2:34201 (local address 127.23.100.1:34557)
0925 10:42:29.686679 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:29.686718 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 10:42:29.687119 (+ 401us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:42:29.690639 (+ 3520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:42:29.690650 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:42:29.691358 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:42:29.692014 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:29.692030 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:29.693835 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:29.693838 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:29.695260 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:29.695271 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:29.696234 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:29.696242 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:29.696558 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:29.697369 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:42:29.697385 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:42:29.700012 (+ 2627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:42:29.703607 (+ 3595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:42:29.703613 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:42:29.703617 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:42:29.704135 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:42:29.704544 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:42:29.704549 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:42:29.704554 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:42:29.704690 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:42:29.705342 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:42:29.705347 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:42:29.705476 (+ 129us) client_negotiation.cc:769] Sending connection context
0925 10:42:29.705960 (+ 484us) client_negotiation.cc:240] Negotiation successful
0925 10:42:29.706203 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":178}
I20260925 10:42:29.706871 6901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:29.687554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:46855 (local address 127.23.100.3:46473)
0925 10:42:29.688362 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:29.688376 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:29.688420 (+ 44us) server_negotiation.cc:407] Connection header received
0925 10:42:29.689847 (+ 1427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:29.689854 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:29.690186 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:29.690522 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:42:29.692203 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:29.693662 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:29.695402 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:29.696005 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:29.696321 (+ 316us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:29.699978 (+ 3657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:42:29.699985 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:42:29.699997 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:42:29.700029 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:42:29.702707 (+ 2678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:29.703383 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:29.703391 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:29.703404 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:29.703631 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:29.704245 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:29.704250 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:29.704253 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:29.704710 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:42:29.704906 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:29.705626 (+ 720us) server_negotiation.cc:299] Negotiation successful
0925 10:42:29.705994 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":414,"threads_started":1}
I20260925 10:42:29.707005 6900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:29.686837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:34557 (local address 127.23.100.2:34201)
0925 10:42:29.688434 (+ 1597us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:29.688444 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:29.688473 (+ 29us) server_negotiation.cc:407] Connection header received
0925 10:42:29.689837 (+ 1364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:29.689845 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:29.690186 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:29.690523 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:42:29.692200 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:29.693647 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:29.695393 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:29.695993 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:29.696194 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:29.700262 (+ 4068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:42:29.700269 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:42:29.700279 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:42:29.700310 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:42:29.703478 (+ 3168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:29.704275 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:29.704286 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:29.704291 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:29.704432 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:29.704793 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:29.704800 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:29.704803 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:29.705219 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:42:29.705403 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:29.706179 (+ 776us) server_negotiation.cc:299] Negotiation successful
0925 10:42:29.706573 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1463,"thread_start_us":1372,"threads_started":1}
W20260925 10:42:29.709492 6516 leader_election.cc:341] T 63f331376be845c9b84765336e7d6553 P fbac24feb0f74a2583264578c3dcf2b3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 899b0712f9d24f4a9a2e0d518962620d (127.23.100.2:34201): Not found: Tablet not found: 63f331376be845c9b84765336e7d6553
W20260925 10:42:29.709728 6517 leader_election.cc:341] T 63f331376be845c9b84765336e7d6553 P fbac24feb0f74a2583264578c3dcf2b3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b41fee5f629349268798ff60f57a31bb (127.23.100.3:46473): Not found: Tablet not found: 63f331376be845c9b84765336e7d6553
I20260925 10:42:29.709846 6517 leader_election.cc:302] T 63f331376be845c9b84765336e7d6553 P fbac24feb0f74a2583264578c3dcf2b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbac24feb0f74a2583264578c3dcf2b3; no voters: 899b0712f9d24f4a9a2e0d518962620d, b41fee5f629349268798ff60f57a31bb
I20260925 10:42:29.710261 6898 raft_consensus.cc:2726] T 63f331376be845c9b84765336e7d6553 P fbac24feb0f74a2583264578c3dcf2b3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 10:42:29.971726 6902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:29.970865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.2:51259)
0925 10:42:29.971343 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:29.971485 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":247,"threads_started":1}
I20260925 10:42:30.396660 6903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:30.395943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.3:53855)
0925 10:42:30.396385 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:30.396494 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":238,"threads_started":1}
I20260925 10:42:30.533542 6904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:30.532894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.1:43573)
0925 10:42:30.533319 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:30.533392 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":232,"threads_started":1}
I20260925 10:42:30.973368 6905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:30.972644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.2:41075)
0925 10:42:30.973102 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:30.973199 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":226,"threads_started":1}
I20260925 10:42:31.380528 6906 raft_consensus.cc:482] T 63f331376be845c9b84765336e7d6553 P fbac24feb0f74a2583264578c3dcf2b3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:42:31.380744 6906 raft_consensus.cc:504] T 63f331376be845c9b84765336e7d6553 P fbac24feb0f74a2583264578c3dcf2b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "899b0712f9d24f4a9a2e0d518962620d" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 34201 } } peers { permanent_uuid: "fbac24feb0f74a2583264578c3dcf2b3" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 36355 } } peers { permanent_uuid: "b41fee5f629349268798ff60f57a31bb" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 46473 } }
I20260925 10:42:31.381397 6906 leader_election.cc:288] T 63f331376be845c9b84765336e7d6553 P fbac24feb0f74a2583264578c3dcf2b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 899b0712f9d24f4a9a2e0d518962620d (127.23.100.2:34201), b41fee5f629349268798ff60f57a31bb (127.23.100.3:46473)
W20260925 10:42:31.382294 6516 leader_election.cc:341] T 63f331376be845c9b84765336e7d6553 P fbac24feb0f74a2583264578c3dcf2b3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 899b0712f9d24f4a9a2e0d518962620d (127.23.100.2:34201): Not found: Tablet not found: 63f331376be845c9b84765336e7d6553
W20260925 10:42:31.382504 6517 leader_election.cc:341] T 63f331376be845c9b84765336e7d6553 P fbac24feb0f74a2583264578c3dcf2b3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b41fee5f629349268798ff60f57a31bb (127.23.100.3:46473): Not found: Tablet not found: 63f331376be845c9b84765336e7d6553
I20260925 10:42:31.382625 6517 leader_election.cc:302] T 63f331376be845c9b84765336e7d6553 P fbac24feb0f74a2583264578c3dcf2b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbac24feb0f74a2583264578c3dcf2b3; no voters: 899b0712f9d24f4a9a2e0d518962620d, b41fee5f629349268798ff60f57a31bb
I20260925 10:42:31.382831 6906 raft_consensus.cc:2726] T 63f331376be845c9b84765336e7d6553 P fbac24feb0f74a2583264578c3dcf2b3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 10:42:31.398141 6907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:31.397360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.3:53007)
0925 10:42:31.397866 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:31.397937 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":272,"threads_started":1}
I20260925 10:42:31.535192 6908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:31.534419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.1:34765)
0925 10:42:31.534925 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:31.535021 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":229,"threads_started":1}
I20260925 10:42:31.975005 6909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:31.974288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.2:47565)
0925 10:42:31.974767 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:31.974847 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":222,"threads_started":1}
I20260925 10:42:32.399544 6910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:32.398891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.3:51345)
0925 10:42:32.399281 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:32.399370 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":208,"threads_started":1}
I20260925 10:42:32.536713 6911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:32.535973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.1:42725)
0925 10:42:32.536485 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:32.536561 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":305,"threads_started":1}
I20260925 10:42:32.976480 6912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:32.975808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.2:41751)
0925 10:42:32.976240 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:32.976316 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":234,"threads_started":1}
I20260925 10:42:33.268582 6913 raft_consensus.cc:482] T 63f331376be845c9b84765336e7d6553 P fbac24feb0f74a2583264578c3dcf2b3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:42:33.268824 6913 raft_consensus.cc:504] T 63f331376be845c9b84765336e7d6553 P fbac24feb0f74a2583264578c3dcf2b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "899b0712f9d24f4a9a2e0d518962620d" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 34201 } } peers { permanent_uuid: "fbac24feb0f74a2583264578c3dcf2b3" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 36355 } } peers { permanent_uuid: "b41fee5f629349268798ff60f57a31bb" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 46473 } }
I20260925 10:42:33.269559 6913 leader_election.cc:288] T 63f331376be845c9b84765336e7d6553 P fbac24feb0f74a2583264578c3dcf2b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 899b0712f9d24f4a9a2e0d518962620d (127.23.100.2:34201), b41fee5f629349268798ff60f57a31bb (127.23.100.3:46473)
W20260925 10:42:33.270529 6516 leader_election.cc:341] T 63f331376be845c9b84765336e7d6553 P fbac24feb0f74a2583264578c3dcf2b3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 899b0712f9d24f4a9a2e0d518962620d (127.23.100.2:34201): Not found: Tablet not found: 63f331376be845c9b84765336e7d6553
W20260925 10:42:33.270748 6517 leader_election.cc:341] T 63f331376be845c9b84765336e7d6553 P fbac24feb0f74a2583264578c3dcf2b3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b41fee5f629349268798ff60f57a31bb (127.23.100.3:46473): Not found: Tablet not found: 63f331376be845c9b84765336e7d6553
I20260925 10:42:33.270876 6517 leader_election.cc:302] T 63f331376be845c9b84765336e7d6553 P fbac24feb0f74a2583264578c3dcf2b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbac24feb0f74a2583264578c3dcf2b3; no voters: 899b0712f9d24f4a9a2e0d518962620d, b41fee5f629349268798ff60f57a31bb
I20260925 10:42:33.271119 6913 raft_consensus.cc:2726] T 63f331376be845c9b84765336e7d6553 P fbac24feb0f74a2583264578c3dcf2b3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 10:42:33.400985 6914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:33.400314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.3:51231)
0925 10:42:33.400739 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:33.400815 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":206,"threads_started":1}
I20260925 10:42:33.538430 6915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:33.537663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.1:36899)
0925 10:42:33.538098 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:33.538203 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":208,"threads_started":1}
I20260925 10:42:33.978091 6916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:33.977440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.2:40431)
0925 10:42:33.977862 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:33.977937 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":213,"threads_started":1}
I20260925 10:42:34.402446 6917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:34.401758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.3:37883)
0925 10:42:34.402155 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:34.402238 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":208,"threads_started":1}
I20260925 10:42:34.539930 6918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:34.539232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.1:44073)
0925 10:42:34.539695 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:34.539772 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":264,"threads_started":1}
I20260925 10:42:34.979661 6919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:34.978947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.2:37445)
0925 10:42:34.979408 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:34.979483 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":259,"threads_started":1}
I20260925 10:42:35.404397 6920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:35.403331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.3:55979)
0925 10:42:35.403960 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:35.404112 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":339,"threads_started":1}
I20260925 10:42:35.541407 6921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:35.540752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.1:39375)
0925 10:42:35.541148 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:35.541226 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":206,"threads_started":1}
I20260925 10:42:35.981251 6922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:35.980517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.2:46297)
0925 10:42:35.980980 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:35.981064 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":214,"threads_started":1}
I20260925 10:42:36.406076 6923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:36.405315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.3:43395)
0925 10:42:36.405807 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:36.405881 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":257,"threads_started":1}
I20260925 10:42:36.542960 6924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:36.542206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.1:58193)
0925 10:42:36.542718 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:36.542805 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":283,"threads_started":1}
I20260925 10:42:36.983013 6925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:36.982224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.2:55445)
0925 10:42:36.982724 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:36.982825 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":290,"threads_started":1}
I20260925 10:42:37.407564 6926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:37.406897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.3:58153)
0925 10:42:37.407307 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:37.407384 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":215,"threads_started":1}
I20260925 10:42:37.544566 6927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:37.543832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.1:48249)
0925 10:42:37.544237 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:37.544316 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":213,"threads_started":1}
I20260925 10:42:37.984491 6928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:37.983834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.2:35145)
0925 10:42:37.984267 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:37.984351 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":213,"threads_started":1}
I20260925 10:42:38.293967 6929 raft_consensus.cc:482] T 63f331376be845c9b84765336e7d6553 P fbac24feb0f74a2583264578c3dcf2b3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:42:38.294267 6929 raft_consensus.cc:504] T 63f331376be845c9b84765336e7d6553 P fbac24feb0f74a2583264578c3dcf2b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "899b0712f9d24f4a9a2e0d518962620d" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 34201 } } peers { permanent_uuid: "fbac24feb0f74a2583264578c3dcf2b3" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 36355 } } peers { permanent_uuid: "b41fee5f629349268798ff60f57a31bb" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 46473 } }
I20260925 10:42:38.294883 6929 leader_election.cc:288] T 63f331376be845c9b84765336e7d6553 P fbac24feb0f74a2583264578c3dcf2b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 899b0712f9d24f4a9a2e0d518962620d (127.23.100.2:34201), b41fee5f629349268798ff60f57a31bb (127.23.100.3:46473)
W20260925 10:42:38.295885 6516 leader_election.cc:341] T 63f331376be845c9b84765336e7d6553 P fbac24feb0f74a2583264578c3dcf2b3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 899b0712f9d24f4a9a2e0d518962620d (127.23.100.2:34201): Not found: Tablet not found: 63f331376be845c9b84765336e7d6553
W20260925 10:42:38.296062 6517 leader_election.cc:341] T 63f331376be845c9b84765336e7d6553 P fbac24feb0f74a2583264578c3dcf2b3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b41fee5f629349268798ff60f57a31bb (127.23.100.3:46473): Not found: Tablet not found: 63f331376be845c9b84765336e7d6553
I20260925 10:42:38.296233 6517 leader_election.cc:302] T 63f331376be845c9b84765336e7d6553 P fbac24feb0f74a2583264578c3dcf2b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbac24feb0f74a2583264578c3dcf2b3; no voters: 899b0712f9d24f4a9a2e0d518962620d, b41fee5f629349268798ff60f57a31bb
I20260925 10:42:38.296474 6929 raft_consensus.cc:2726] T 63f331376be845c9b84765336e7d6553 P fbac24feb0f74a2583264578c3dcf2b3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 10:42:38.409034 6930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:38.408367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.3:39937)
0925 10:42:38.408802 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:38.408876 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":205,"threads_started":1}
I20260925 10:42:38.546552 6931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:38.545480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.1:53727)
0925 10:42:38.546068 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:38.546227 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":288,"threads_started":1}
I20260925 10:42:38.986132 6932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:38.985489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.2:43179)
0925 10:42:38.985882 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:38.985962 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":214,"threads_started":1}
I20260925 10:42:39.410670 6933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:39.409944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.3:33699)
0925 10:42:39.410410 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:39.410496 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":222,"threads_started":1}
I20260925 10:42:39.547981 6934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:39.547332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.1:37825)
0925 10:42:39.547740 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:39.547812 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":216,"threads_started":1}
I20260925 10:42:39.987782 6935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:39.987038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.2:59797)
0925 10:42:39.987554 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:39.987635 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":280,"threads_started":1}
I20260925 10:42:40.412092 6936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:40.411448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.3:55007)
0925 10:42:40.411872 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:40.411949 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":218,"threads_started":1}
I20260925 10:42:40.549399 6937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:40.548734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.1:51353)
0925 10:42:40.549151 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:40.549227 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":224,"threads_started":1}
I20260925 10:42:40.989500 6938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:40.988720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.2:48965)
0925 10:42:40.989266 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:40.989343 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":315,"threads_started":1}
I20260925 10:42:41.413611 6939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:41.412973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.3:57411)
0925 10:42:41.413369 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:41.413452 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":204,"threads_started":1}
I20260925 10:42:41.551105 6940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:41.550406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.1:54803)
0925 10:42:41.550840 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:41.550918 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":239,"threads_started":1}
I20260925 10:42:41.991061 6941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:41.990387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.2:37903)
0925 10:42:41.990821 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:41.990905 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":231,"threads_started":1}
I20260925 10:42:42.415174 6942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:42.414456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.3:34465)
0925 10:42:42.414926 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:42.415025 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":250,"threads_started":1}
I20260925 10:42:42.552572 6943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:42.551899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.1:48625)
0925 10:42:42.552278 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:42.552350 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":205,"threads_started":1}
I20260925 10:42:42.988147 6944 raft_consensus.cc:482] T 63f331376be845c9b84765336e7d6553 P fbac24feb0f74a2583264578c3dcf2b3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:42:42.988366 6944 raft_consensus.cc:504] T 63f331376be845c9b84765336e7d6553 P fbac24feb0f74a2583264578c3dcf2b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "899b0712f9d24f4a9a2e0d518962620d" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 34201 } } peers { permanent_uuid: "fbac24feb0f74a2583264578c3dcf2b3" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 36355 } } peers { permanent_uuid: "b41fee5f629349268798ff60f57a31bb" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 46473 } }
I20260925 10:42:42.988979 6944 leader_election.cc:288] T 63f331376be845c9b84765336e7d6553 P fbac24feb0f74a2583264578c3dcf2b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 899b0712f9d24f4a9a2e0d518962620d (127.23.100.2:34201), b41fee5f629349268798ff60f57a31bb (127.23.100.3:46473)
W20260925 10:42:42.989919 6516 leader_election.cc:341] T 63f331376be845c9b84765336e7d6553 P fbac24feb0f74a2583264578c3dcf2b3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 899b0712f9d24f4a9a2e0d518962620d (127.23.100.2:34201): Not found: Tablet not found: 63f331376be845c9b84765336e7d6553
W20260925 10:42:42.990144 6517 leader_election.cc:341] T 63f331376be845c9b84765336e7d6553 P fbac24feb0f74a2583264578c3dcf2b3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b41fee5f629349268798ff60f57a31bb (127.23.100.3:46473): Not found: Tablet not found: 63f331376be845c9b84765336e7d6553
I20260925 10:42:42.990283 6517 leader_election.cc:302] T 63f331376be845c9b84765336e7d6553 P fbac24feb0f74a2583264578c3dcf2b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbac24feb0f74a2583264578c3dcf2b3; no voters: 899b0712f9d24f4a9a2e0d518962620d, b41fee5f629349268798ff60f57a31bb
I20260925 10:42:42.990674 6944 raft_consensus.cc:2726] T 63f331376be845c9b84765336e7d6553 P fbac24feb0f74a2583264578c3dcf2b3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 10:42:42.992708 6945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:42.991821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.2:40689)
0925 10:42:42.992406 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:42.992513 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":295,"threads_started":1}
I20260925 10:42:43.416729 6946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:43.416050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.3:57027)
0925 10:42:43.416459 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:43.416548 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":225,"threads_started":1}
I20260925 10:42:43.554536 6947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:43.553640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.1:45121)
0925 10:42:43.554180 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:43.554296 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":282,"threads_started":1}
I20260925 10:42:43.994832 6948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:43.994042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.2:36117)
0925 10:42:43.994588 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:43.994679 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":330,"threads_started":1}
I20260925 10:42:44.400373 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 6466
I20260925 10:42:44.418783 6950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:44.417822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44293 (local address 127.23.100.3:38905)
0925 10:42:44.418441 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:44.418589 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":295,"threads_started":1}
I20260925 10:42:44.422431 6949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:44.401472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50238 (local address 127.23.100.1:36355)
0925 10:42:44.402046 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:44.402053 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:44.402105 (+ 52us) server_negotiation.cc:407] Connection header received
0925 10:42:44.402179 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:44.402186 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:44.402532 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:44.402723 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:42:44.404830 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:44.405799 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:44.407164 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:44.407726 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:44.407867 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:44.416113 (+ 8246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:42:44.416125 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:42:44.416138 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:42:44.416172 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:42:44.419809 (+ 3637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:44.420451 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:44.420458 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:44.420460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:44.420583 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:44.420902 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:44.420905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:44.420907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:44.421236 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:42:44.421474 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:44.421757 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 10:42:44.422057 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":254,"threads_started":1}
I20260925 10:42:44.423657 6625 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:42:44.511705 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 6466
I20260925 10:42:44.529243 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 6633
I20260925 10:42:44.542095 6952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:44.530004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46834 (local address 127.23.100.2:34201)
0925 10:42:44.530509 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:44.530514 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:44.530530 (+ 16us) server_negotiation.cc:407] Connection header received
0925 10:42:44.530583 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:44.530586 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:44.530643 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:44.530729 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:42:44.532008 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:44.532825 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:44.534006 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:44.534499 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:44.534594 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:44.537063 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:42:44.537067 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:42:44.537070 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:42:44.537098 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:42:44.539849 (+ 2751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:44.540426 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:44.540430 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:44.540432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:44.540561 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:44.540878 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:44.540880 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:44.540882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:44.541098 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:42:44.541269 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:44.541513 (+ 244us) server_negotiation.cc:299] Negotiation successful
0925 10:42:44.541671 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":300,"threads_started":1}
I20260925 10:42:44.542856 6757 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:42:44.625098 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 6633
I20260925 10:42:44.643035 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 6765
I20260925 10:42:44.655834 6954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:44.643822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48598 (local address 127.23.100.3:46473)
0925 10:42:44.644338 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:44.644343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:44.644361 (+ 18us) server_negotiation.cc:407] Connection header received
0925 10:42:44.644417 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:44.644421 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:44.644488 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:44.644584 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:42:44.645911 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:44.646846 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:44.648040 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:44.648444 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:44.648535 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:44.651106 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:42:44.651110 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:42:44.651114 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:42:44.651143 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:42:44.653587 (+ 2444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:44.654202 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:44.654206 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:44.654208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:44.654400 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:44.654708 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:44.654711 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:44.654713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:44.654957 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:42:44.655155 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:44.655396 (+ 241us) server_negotiation.cc:299] Negotiation successful
0925 10:42:44.655550 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":253,"threads_started":1}
I20260925 10:42:44.656540 6888 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:42:44.739017 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 6765
2026-09-25T10:42:44Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (23922 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:42:44 dist-test-slave-dxdd krb5kdc[6958](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:42:44 dist-test-slave-dxdd krb5kdc[6958](info): set up 2 sockets
Sep 25 10:42:44 dist-test-slave-dxdd krb5kdc[6958](info): commencing operation
krb5kdc: starting...
W20260925 10:42:45.843034 23952 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.005s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Sep 25 10:42:45 dist-test-slave-dxdd krb5kdc[6958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:42:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:42:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
I20260925 10:42:46.094405 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:35471
--webserver_interface=127.23.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:38443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.62:35471
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:42:46.413455 6971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:42:46.413800 6971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:42:46.413967 6971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:42:46.421144 6971 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 10:42:46.421273 6971 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 10:42:46.421382 6971 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 10:42:46.421495 6971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:42:46.421593 6971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:42:46.421695 6971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:42:46.421765 6971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:42:46.421844 6971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:42:46.432415 6971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:38443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.23.100.62:35471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:35471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:42:46.433705 6971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:42:46.435127 6971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:42:46.447063 6976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:42:46.447034 6979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:42:46.447120 6977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:42:46.447626 6971 server_base.cc:1034] running on GCE node
I20260925 10:42:46.448392 6971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:42:46.449653 6971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:42:46.450860 6971 hybrid_clock.cc:648] HybridClock initialized: now 1790332966450779 us; error 54 us; skew 500 ppm
I20260925 10:42:46.451195 6971 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:42:46 dist-test-slave-dxdd krb5kdc[6958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332966, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:42:46.454567 6971 init.cc:377] Logged in from keytab as kudu/127.23.100.62@KRBTEST.COM (short username kudu)
I20260925 10:42:46.456084 6971 webserver.cc:460] Webserver started at http://127.23.100.62:46277/ using document root <none> and password file <none>
I20260925 10:42:46.456768 6971 fs_manager.cc:359] Metadata directory not provided
I20260925 10:42:46.456851 6971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:42:46.457098 6971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:42:46.459971 6971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/master-0/data/instance:
uuid: "282881ffd90e4e42a826159c2fe1678d"
format_stamp: "Formatted at 2026-09-25 10:42:46 on dist-test-slave-dxdd"
server_key: "6d6e6f0211224999f567176237198484"
server_key_iv: "04f228acc47f06b9b946e5ed5ad5c5ee"
server_key_version: "encryptionkey@0"
I20260925 10:42:46.460659 6971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/master-0/wal/instance:
uuid: "282881ffd90e4e42a826159c2fe1678d"
format_stamp: "Formatted at 2026-09-25 10:42:46 on dist-test-slave-dxdd"
server_key: "6d6e6f0211224999f567176237198484"
server_key_iv: "04f228acc47f06b9b946e5ed5ad5c5ee"
server_key_version: "encryptionkey@0"
I20260925 10:42:46.465412 6971 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 10:42:46.468688 6986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:42:46.469893 6971 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 10:42:46.470049 6971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/master-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/master-0/wal
uuid: "282881ffd90e4e42a826159c2fe1678d"
format_stamp: "Formatted at 2026-09-25 10:42:46 on dist-test-slave-dxdd"
server_key: "6d6e6f0211224999f567176237198484"
server_key_iv: "04f228acc47f06b9b946e5ed5ad5c5ee"
server_key_version: "encryptionkey@0"
I20260925 10:42:46.470249 6971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:42:46.492357 6971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:42:46.496977 6971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:42:46.497253 6971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:42:46.514997 6971 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.62:35471
I20260925 10:42:46.515017 7037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.62:35471 every 8 connection(s)
I20260925 10:42:46.516281 6971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
I20260925 10:42:46.517601 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 6971
I20260925 10:42:46.517804 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/master-0/wal/instance
I20260925 10:42:46.518498 23952 external_mini_cluster.cc:1428] Setting key 474445283b0863b3df4d3d481d33aeae
I20260925 10:42:46.520602 7038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 10:42:46 dist-test-slave-dxdd krb5kdc[6958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:42:46.533486 7038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d: Bootstrap starting.
I20260925 10:42:46.537505 7038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d: Neither blocks nor log segments found. Creating new log.
I20260925 10:42:46.538849 7038 log.cc:824] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d: Log is configured to *not* fsync() on all Append() calls
I20260925 10:42:46.542080 7038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d: No bootstrap required, opened a new log
I20260925 10:42:46.542757 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:46.520469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58678 (local address 127.23.100.62:35471)
0925 10:42:46.521301 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:46.521316 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:46.521364 (+ 48us) server_negotiation.cc:407] Connection header received
0925 10:42:46.523641 (+ 2277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:46.523666 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:46.524152 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:46.524587 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:42:46.525857 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:46.527934 (+ 2077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:46.529258 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:46.530220 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:46.530452 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:46.534420 (+ 3968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:42:46.534432 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:42:46.534448 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:42:46.534496 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:42:46.538442 (+ 3946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:46.539107 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:46.539115 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:46.539122 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:46.539353 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:46.539814 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:46.539818 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:46.539820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:46.540423 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:42:46.540693 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:46.541052 (+ 359us) server_negotiation.cc:299] Negotiation successful
0925 10:42:46.541467 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":359,"threads_started":1}
I20260925 10:42:46.550951 7038 raft_consensus.cc:348] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "282881ffd90e4e42a826159c2fe1678d" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 35471 } }
I20260925 10:42:46.551288 7038 raft_consensus.cc:374] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:42:46.551362 7038 raft_consensus.cc:729] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 282881ffd90e4e42a826159c2fe1678d, State: Initialized, Role: FOLLOWER
I20260925 10:42:46.551808 7038 consensus_queue.cc:260] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "282881ffd90e4e42a826159c2fe1678d" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 35471 } }
I20260925 10:42:46.552026 7038 raft_consensus.cc:388] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:42:46.552196 7038 raft_consensus.cc:482] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:42:46.552374 7038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:42:46.554483 7038 raft_consensus.cc:504] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "282881ffd90e4e42a826159c2fe1678d" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 35471 } }
I20260925 10:42:46.554950 7038 leader_election.cc:302] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 282881ffd90e4e42a826159c2fe1678d; no voters:
I20260925 10:42:46.555486 7038 leader_election.cc:288] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:42:46.555619 7043 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:42:46.556821 7043 raft_consensus.cc:686] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d [term 1 LEADER]: Becoming Leader. State: Replica: 282881ffd90e4e42a826159c2fe1678d, State: Running, Role: LEADER
I20260925 10:42:46.557292 7043 consensus_queue.cc:237] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "282881ffd90e4e42a826159c2fe1678d" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 35471 } }
I20260925 10:42:46.557533 7038 sys_catalog.cc:564] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:42:46.560113 7045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 282881ffd90e4e42a826159c2fe1678d. Latest consensus state: current_term: 1 leader_uuid: "282881ffd90e4e42a826159c2fe1678d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "282881ffd90e4e42a826159c2fe1678d" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 35471 } } }
I20260925 10:42:46.560631 7045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d [sys.catalog]: This master's current role is: LEADER
I20260925 10:42:46.560521 7044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "282881ffd90e4e42a826159c2fe1678d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "282881ffd90e4e42a826159c2fe1678d" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 35471 } } }
I20260925 10:42:46.560855 7044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d [sys.catalog]: This master's current role is: LEADER
I20260925 10:42:46.561663 7052 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:42:46.566650 7052 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:42:46.575109 7052 catalog_manager.cc:1347] Generated new cluster ID: 78d8f9c6914141d38315fd6f65ef128f
I20260925 10:42:46.575222 7052 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:42:46.590188 7052 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:42:46.591560 7052 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:42:46.600229 7052 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d: Generated new TSK 0
I20260925 10:42:46.601038 7052 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab.
Entry for principal kudu/127.23.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab.
I20260925 10:42:46.661520 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.1:0
--local_ip_for_outbound_sockets=127.23.100.1
--webserver_interface=127.23.100.1
--webserver_port=0
--tserver_master_addrs=127.23.100.62:35471
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab
--principal=kudu/127.23.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:38443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:42:46.980131 7064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:42:46.980391 7064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:42:46.980558 7064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:42:46.987412 7064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:42:46.987519 7064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:42:46.987744 7064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.1
I20260925 10:42:46.997628 7064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:38443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab
--principal=kudu/127.23.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.1
--webserver_port=0
--tserver_master_addrs=127.23.100.62:35471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.1
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:42:46.998915 7064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:42:47.000211 7064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:42:47.012843 7070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:42:47.012904 7072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:42:47.012799 7069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:42:47.013703 7064 server_base.cc:1034] running on GCE node
I20260925 10:42:47.014220 7064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:42:47.015105 7064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:42:47.016305 7064 hybrid_clock.cc:648] HybridClock initialized: now 1790332967016275 us; error 37 us; skew 500 ppm
I20260925 10:42:47.016564 7064 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:42:47 dist-test-slave-dxdd krb5kdc[6958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332967, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:42:47.019497 7064 init.cc:377] Logged in from keytab as kudu/127.23.100.1@KRBTEST.COM (short username kudu)
I20260925 10:42:47.020815 7064 webserver.cc:460] Webserver started at http://127.23.100.1:40453/ using document root <none> and password file <none>
I20260925 10:42:47.021417 7064 fs_manager.cc:359] Metadata directory not provided
I20260925 10:42:47.021512 7064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:42:47.021750 7064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:42:47.024528 7064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-0/data/instance:
uuid: "24c5c47272fe4d52b77ba1edc0d1e120"
format_stamp: "Formatted at 2026-09-25 10:42:47 on dist-test-slave-dxdd"
server_key: "aa9594b95baee257d2bd937db21816b5"
server_key_iv: "c678bb0bf00dafa9173c703922bdfd3c"
server_key_version: "encryptionkey@0"
I20260925 10:42:47.025200 7064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-0/wal/instance:
uuid: "24c5c47272fe4d52b77ba1edc0d1e120"
format_stamp: "Formatted at 2026-09-25 10:42:47 on dist-test-slave-dxdd"
server_key: "aa9594b95baee257d2bd937db21816b5"
server_key_iv: "c678bb0bf00dafa9173c703922bdfd3c"
server_key_version: "encryptionkey@0"
I20260925 10:42:47.030145 7064 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 10:42:47.033419 7079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:42:47.034759 7064 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:42:47.034919 7064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-0/wal
uuid: "24c5c47272fe4d52b77ba1edc0d1e120"
format_stamp: "Formatted at 2026-09-25 10:42:47 on dist-test-slave-dxdd"
server_key: "aa9594b95baee257d2bd937db21816b5"
server_key_iv: "c678bb0bf00dafa9173c703922bdfd3c"
server_key_version: "encryptionkey@0"
I20260925 10:42:47.035120 7064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:42:47.056293 7064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:42:47.060418 7064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:42:47.060702 7064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:42:47.061647 7064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:42:47.062960 7064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:42:47.063031 7064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:42:47.063202 7064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:42:47.063256 7064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:42:47.093894 7064 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.1:43897
I20260925 10:42:47.093931 7191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.1:43897 every 8 connection(s)
I20260925 10:42:47.095196 7064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
I20260925 10:42:47.102406 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 7064
I20260925 10:42:47.102604 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-0/wal/instance
I20260925 10:42:47.103255 23952 external_mini_cluster.cc:1428] Setting key 80bfbe937184c87df897b95798323c9f
Sep 25 10:42:47 dist-test-slave-dxdd krb5kdc[6958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332967, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.1@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:42:47.114931 7194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:47.097689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:52073 (local address 127.23.100.62:35471)
0925 10:42:47.098111 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:47.098115 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:47.099087 (+ 972us) server_negotiation.cc:407] Connection header received
0925 10:42:47.101105 (+ 2018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:47.101111 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:47.101195 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:47.101319 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:42:47.102917 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:47.103666 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:47.105437 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:47.105815 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:47.105947 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:47.109264 (+ 3317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:42:47.109269 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:42:47.109272 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:42:47.109301 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:42:47.111846 (+ 2545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:47.112698 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:47.112702 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:47.112704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:47.112820 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:47.113291 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:47.113294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:47.113296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:47.113534 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:42:47.113641 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:47.114516 (+ 875us) server_negotiation.cc:299] Negotiation successful
0925 10:42:47.114676 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":217,"threads_started":1}
I20260925 10:42:47.116338 7195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:47.098231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.1:52073)
0925 10:42:47.098935 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:47.098971 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 10:42:47.100852 (+ 1881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:42:47.101559 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:42:47.101566 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:42:47.102041 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:42:47.102754 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:47.102773 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:47.103816 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:47.103821 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:47.105297 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:47.105308 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:47.106035 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:47.106043 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:47.106269 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:47.106962 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:42:47.106988 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:42:47.109053 (+ 2065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:42:47.111986 (+ 2933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:42:47.111995 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:42:47.112010 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:42:47.112554 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:42:47.112946 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:42:47.112953 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:42:47.112959 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:42:47.113184 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:42:47.113653 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:42:47.113663 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:42:47.114082 (+ 419us) client_negotiation.cc:769] Sending connection context
0925 10:42:47.114401 (+ 319us) client_negotiation.cc:240] Negotiation successful
0925 10:42:47.114952 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":251,"threads_started":1}
I20260925 10:42:47.117902 7192 heartbeater.cc:347] Connected to a master server at 127.23.100.62:35471
I20260925 10:42:47.118191 7192 heartbeater.cc:464] Registering TS with master...
I20260925 10:42:47.118942 7192 heartbeater.cc:511] Master 127.23.100.62:35471 requested a full tablet report, sending...
I20260925 10:42:47.120939 7003 ts_manager.cc:194] Registered new tserver with Master: 24c5c47272fe4d52b77ba1edc0d1e120 (127.23.100.1:43897)
I20260925 10:42:47.122409 7003 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.1@KRBTEST.COM'} at 127.23.100.1:52073
WARNING: no policy specified for kudu/127.23.100.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab.
Entry for principal kudu/127.23.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab.
I20260925 10:42:47.157332 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.2:0
--local_ip_for_outbound_sockets=127.23.100.2
--webserver_interface=127.23.100.2
--webserver_port=0
--tserver_master_addrs=127.23.100.62:35471
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab
--principal=kudu/127.23.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:38443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:42:47.472374 7198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:42:47.472688 7198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:42:47.472855 7198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:42:47.480335 7198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:42:47.480495 7198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:42:47.480687 7198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.2
I20260925 10:42:47.490276 7198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:38443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab
--principal=kudu/127.23.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.2
--webserver_port=0
--tserver_master_addrs=127.23.100.62:35471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.2
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:42:47.491554 7198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:42:47.492863 7198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:42:47.505729 7203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:42:47.505697 7204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:42:47.505787 7206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:42:47.506588 7198 server_base.cc:1034] running on GCE node
I20260925 10:42:47.507126 7198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:42:47.507992 7198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:42:47.509186 7198 hybrid_clock.cc:648] HybridClock initialized: now 1790332967509149 us; error 29 us; skew 500 ppm
I20260925 10:42:47.509505 7198 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:42:47 dist-test-slave-dxdd krb5kdc[6958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332967, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:42:47.512609 7198 init.cc:377] Logged in from keytab as kudu/127.23.100.2@KRBTEST.COM (short username kudu)
I20260925 10:42:47.514019 7198 webserver.cc:460] Webserver started at http://127.23.100.2:42107/ using document root <none> and password file <none>
I20260925 10:42:47.514808 7198 fs_manager.cc:359] Metadata directory not provided
I20260925 10:42:47.514958 7198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:42:47.515252 7198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:42:47.517999 7198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-1/data/instance:
uuid: "bd045bbe55db4881bcd6d93cbcc2a3af"
format_stamp: "Formatted at 2026-09-25 10:42:47 on dist-test-slave-dxdd"
server_key: "b0535cac90591377fca569b392e5d358"
server_key_iv: "b08fc578239d0f300681f7df6f7ad797"
server_key_version: "encryptionkey@0"
I20260925 10:42:47.518760 7198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-1/wal/instance:
uuid: "bd045bbe55db4881bcd6d93cbcc2a3af"
format_stamp: "Formatted at 2026-09-25 10:42:47 on dist-test-slave-dxdd"
server_key: "b0535cac90591377fca569b392e5d358"
server_key_iv: "b08fc578239d0f300681f7df6f7ad797"
server_key_version: "encryptionkey@0"
I20260925 10:42:47.523661 7198 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260925 10:42:47.527012 7213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:42:47.528237 7198 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 10:42:47.528398 7198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-1/wal
uuid: "bd045bbe55db4881bcd6d93cbcc2a3af"
format_stamp: "Formatted at 2026-09-25 10:42:47 on dist-test-slave-dxdd"
server_key: "b0535cac90591377fca569b392e5d358"
server_key_iv: "b08fc578239d0f300681f7df6f7ad797"
server_key_version: "encryptionkey@0"
I20260925 10:42:47.528607 7198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:42:47.543891 7198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:42:47.547577 7198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:42:47.547855 7198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:42:47.548799 7198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:42:47.550076 7198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:42:47.550187 7198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:42:47.550397 7198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:42:47.550479 7198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260925 10:42:47.582465 7198 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.2:39469
I20260925 10:42:47.582497 7325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.2:39469 every 8 connection(s)
I20260925 10:42:47.583756 7198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
I20260925 10:42:47.589241 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 7198
I20260925 10:42:47.589476 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-1/wal/instance
I20260925 10:42:47.589915 23952 external_mini_cluster.cc:1428] Setting key 9a797686ba73395dd68f4399b8cff972
Sep 25 10:42:47 dist-test-slave-dxdd krb5kdc[6958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332967, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.2@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:42:47.606639 7194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:47.586601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:36593 (local address 127.23.100.62:35471)
0925 10:42:47.586780 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:47.586785 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:47.587959 (+ 1174us) server_negotiation.cc:407] Connection header received
0925 10:42:47.589871 (+ 1912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:47.589876 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:47.589975 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:47.590094 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:42:47.591968 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:47.592727 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:47.595273 (+ 2546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:47.595676 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:47.595768 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:47.599807 (+ 4039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:42:47.599811 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:42:47.599814 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:42:47.599846 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:42:47.603328 (+ 3482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:47.604038 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:47.604046 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:47.604053 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:47.604238 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:47.604673 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:47.604681 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:47.604687 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:47.605064 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:42:47.605228 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:47.605880 (+ 652us) server_negotiation.cc:299] Negotiation successful
0925 10:42:47.606140 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 10:42:47.607242 7328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:47.587120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.2:36593)
0925 10:42:47.587800 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:47.587841 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 10:42:47.589610 (+ 1769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:42:47.590409 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:42:47.590421 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:42:47.591132 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:42:47.591802 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:47.591829 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:47.592896 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:47.592904 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:47.595135 (+ 2231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:47.595153 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:47.595949 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:47.595958 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:47.596349 (+ 391us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:47.597149 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:42:47.597171 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:42:47.599568 (+ 2397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:42:47.603471 (+ 3903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:42:47.603478 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:42:47.603491 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:42:47.603898 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:42:47.604362 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:42:47.604365 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:42:47.604368 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:42:47.604554 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:42:47.605189 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:42:47.605194 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:42:47.605513 (+ 319us) client_negotiation.cc:769] Sending connection context
0925 10:42:47.605730 (+ 217us) client_negotiation.cc:240] Negotiation successful
0925 10:42:47.606103 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":273,"threads_started":1}
I20260925 10:42:47.608806 7326 heartbeater.cc:347] Connected to a master server at 127.23.100.62:35471
I20260925 10:42:47.609135 7326 heartbeater.cc:464] Registering TS with master...
I20260925 10:42:47.610023 7326 heartbeater.cc:511] Master 127.23.100.62:35471 requested a full tablet report, sending...
I20260925 10:42:47.611639 7003 ts_manager.cc:194] Registered new tserver with Master: bd045bbe55db4881bcd6d93cbcc2a3af (127.23.100.2:39469)
I20260925 10:42:47.612469 7003 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.2@KRBTEST.COM'} at 127.23.100.2:36593
WARNING: no policy specified for kudu/127.23.100.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab.
Entry for principal kudu/127.23.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab.
I20260925 10:42:47.649124 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.3:0
--local_ip_for_outbound_sockets=127.23.100.3
--webserver_interface=127.23.100.3
--webserver_port=0
--tserver_master_addrs=127.23.100.62:35471
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab
--principal=kudu/127.23.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:38443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:42:47.969130 7331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:42:47.969386 7331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:42:47.969482 7331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:42:47.976480 7331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:42:47.976568 7331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:42:47.976673 7331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.3
I20260925 10:42:47.985630 7331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:38443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab
--principal=kudu/127.23.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.3
--webserver_port=0
--tserver_master_addrs=127.23.100.62:35471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.3
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:42:47.986822 7331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:42:47.987988 7331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:42:48.000869 7339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:42:48.001101 7336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:42:48.000955 7337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:42:48.001758 7331 server_base.cc:1034] running on GCE node
I20260925 10:42:48.002420 7331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:42:48.003334 7331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:42:48.004565 7331 hybrid_clock.cc:648] HybridClock initialized: now 1790332968004523 us; error 52 us; skew 500 ppm
I20260925 10:42:48.004868 7331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:42:48 dist-test-slave-dxdd krb5kdc[6958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332968, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:42:48.007982 7331 init.cc:377] Logged in from keytab as kudu/127.23.100.3@KRBTEST.COM (short username kudu)
I20260925 10:42:48.009429 7331 webserver.cc:460] Webserver started at http://127.23.100.3:40667/ using document root <none> and password file <none>
I20260925 10:42:48.010094 7331 fs_manager.cc:359] Metadata directory not provided
I20260925 10:42:48.010223 7331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:42:48.010557 7331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:42:48.013207 7331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-2/data/instance:
uuid: "938822573aaf469abe7f961a07d8e36b"
format_stamp: "Formatted at 2026-09-25 10:42:48 on dist-test-slave-dxdd"
server_key: "c8114d8716ce625d6069bd7f1d2f73e3"
server_key_iv: "99b959de31f0f6579b68096f16f12e02"
server_key_version: "encryptionkey@0"
I20260925 10:42:48.013873 7331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-2/wal/instance:
uuid: "938822573aaf469abe7f961a07d8e36b"
format_stamp: "Formatted at 2026-09-25 10:42:48 on dist-test-slave-dxdd"
server_key: "c8114d8716ce625d6069bd7f1d2f73e3"
server_key_iv: "99b959de31f0f6579b68096f16f12e02"
server_key_version: "encryptionkey@0"
I20260925 10:42:48.018785 7331 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260925 10:42:48.022035 7346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:42:48.023308 7331 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:42:48.023494 7331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-2/wal
uuid: "938822573aaf469abe7f961a07d8e36b"
format_stamp: "Formatted at 2026-09-25 10:42:48 on dist-test-slave-dxdd"
server_key: "c8114d8716ce625d6069bd7f1d2f73e3"
server_key_iv: "99b959de31f0f6579b68096f16f12e02"
server_key_version: "encryptionkey@0"
I20260925 10:42:48.023689 7331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:42:48.056728 7331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:42:48.060868 7331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:42:48.061151 7331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:42:48.062119 7331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:42:48.063416 7331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:42:48.063485 7331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:42:48.063647 7331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:42:48.063714 7331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:42:48.096714 7331 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.3:44349
I20260925 10:42:48.096680 7458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.3:44349 every 8 connection(s)
I20260925 10:42:48.098217 7331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
I20260925 10:42:48.101636 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 7331
I20260925 10:42:48.101822 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-2/wal/instance
I20260925 10:42:48.102360 23952 external_mini_cluster.cc:1428] Setting key e23b67ad3ce448774a439755370559c9
Sep 25 10:42:48 dist-test-slave-dxdd krb5kdc[6958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332968, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.3@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:42:48.117574 7194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:48.100820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:41819 (local address 127.23.100.62:35471)
0925 10:42:48.100997 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:48.101001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:48.102165 (+ 1164us) server_negotiation.cc:407] Connection header received
0925 10:42:48.104228 (+ 2063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:48.104232 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:48.104295 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:48.104420 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:42:48.105977 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:48.106786 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:48.108305 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:48.108745 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:48.108877 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:48.112094 (+ 3217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:42:48.112099 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:42:48.112102 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:42:48.112133 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:42:48.114696 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:48.115348 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:48.115356 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:48.115373 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:48.115530 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:48.115940 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:48.115947 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:48.115951 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:48.116210 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:42:48.116368 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:48.117009 (+ 641us) server_negotiation.cc:299] Negotiation successful
0925 10:42:48.117212 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 10:42:48.118491 7461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:48.101231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.3:41819)
0925 10:42:48.101997 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:48.102045 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 10:42:48.103936 (+ 1891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:42:48.104645 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:42:48.104652 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:42:48.105281 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:42:48.105807 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:48.105824 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:48.106949 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:48.106953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:48.108163 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:48.108174 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:48.108974 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:48.108982 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:48.109200 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:48.109787 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:42:48.109812 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:42:48.111885 (+ 2073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:42:48.114825 (+ 2940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:42:48.114834 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:42:48.114847 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:42:48.115229 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:42:48.115637 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:42:48.115643 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:42:48.115646 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:42:48.115816 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:42:48.116336 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:42:48.116343 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:42:48.116666 (+ 323us) client_negotiation.cc:769] Sending connection context
0925 10:42:48.116920 (+ 254us) client_negotiation.cc:240] Negotiation successful
0925 10:42:48.117389 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":340,"threads_started":1}
I20260925 10:42:48.119680 7459 heartbeater.cc:347] Connected to a master server at 127.23.100.62:35471
I20260925 10:42:48.119987 7459 heartbeater.cc:464] Registering TS with master...
I20260925 10:42:48.120639 7459 heartbeater.cc:511] Master 127.23.100.62:35471 requested a full tablet report, sending...
I20260925 10:42:48.121934 7003 ts_manager.cc:194] Registered new tserver with Master: 938822573aaf469abe7f961a07d8e36b (127.23.100.3:44349)
I20260925 10:42:48.122757 7003 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.3@KRBTEST.COM'} at 127.23.100.3:41819
I20260925 10:42:48.125451 7192 heartbeater.cc:503] Master 127.23.100.62:35471 was elected leader, sending a full tablet report...
I20260925 10:42:48.129084 23952 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:42:48.145102 7194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:48.133419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58680 (local address 127.23.100.62:35471)
0925 10:42:48.133630 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:48.133637 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:48.133968 (+ 331us) server_negotiation.cc:407] Connection header received
0925 10:42:48.134091 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:48.134094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:48.134163 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:48.134239 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:42:48.135649 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:48.136325 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:48.137647 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:48.138045 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:48.138130 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:48.140573 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:42:48.140578 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:42:48.140581 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:42:48.140609 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:42:48.142970 (+ 2361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:48.143541 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:48.143544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:48.143546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:48.143658 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:48.143959 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:48.143962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:48.143964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:48.144241 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:42:48.144425 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:48.144674 (+ 249us) server_negotiation.cc:299] Negotiation successful
0925 10:42:48.144818 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 10:42:48.148821 7003 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58680:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:42:48.155418 7003 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:42:48.178732 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:48.167001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50610 (local address 127.23.100.3:44349)
0925 10:42:48.170263 (+ 3262us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:48.170271 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:48.170333 (+ 62us) server_negotiation.cc:407] Connection header received
0925 10:42:48.170448 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:48.170455 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:48.170718 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:48.171771 (+ 1053us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:42:48.172065 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:48.173581 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:48.176647 (+ 3066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:48.177594 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:48.177765 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:48.178001 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:48.178247 (+ 246us) server_negotiation.cc:299] Negotiation successful
0925 10:42:48.178416 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3077,"thread_start_us":2915,"threads_started":1}
I20260925 10:42:48.178833 7471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:48.166866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.3:44349 (local address 127.0.0.1:50610)
0925 10:42:48.167903 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:48.167954 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 10:42:48.168146 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:42:48.170972 (+ 2826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:42:48.170982 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:42:48.171018 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:42:48.171557 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:48.171574 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:48.173732 (+ 2158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:48.173744 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:48.176446 (+ 2702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:48.176460 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:48.177791 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:48.177796 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:48.177989 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:48.178045 (+ 56us) client_negotiation.cc:769] Sending connection context
0925 10:42:48.178201 (+ 156us) client_negotiation.cc:240] Negotiation successful
0925 10:42:48.178398 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":780,"mutex_wait_us":224,"thread_start_us":1486,"threads_started":1}
I20260925 10:42:48.179524 7474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:48.167402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:43897 (local address 127.0.0.1:55138)
0925 10:42:48.170340 (+ 2938us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:48.170357 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:42:48.170500 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:42:48.171059 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:42:48.171076 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:42:48.171110 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:42:48.171577 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:48.171585 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:48.173774 (+ 2189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:48.173779 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:48.176357 (+ 2578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:48.176373 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:48.178469 (+ 2096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:48.178481 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:48.178721 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:48.178858 (+ 137us) client_negotiation.cc:769] Sending connection context
0925 10:42:48.178937 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 10:42:48.179053 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2756,"mutex_wait_us":248,"thread_start_us":3378,"threads_started":1}
I20260925 10:42:48.180032 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:48.168138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48536 (local address 127.23.100.2:39469)
0925 10:42:48.169369 (+ 1231us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:48.169380 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:48.171162 (+ 1782us) server_negotiation.cc:407] Connection header received
0925 10:42:48.171441 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:48.171450 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:48.171679 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:48.171850 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:42:48.172400 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:48.174055 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:48.176231 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:48.177945 (+ 1714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:48.178096 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:48.178264 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:48.179589 (+ 1325us) server_negotiation.cc:299] Negotiation successful
0925 10:42:48.179744 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1042,"thread_start_us":920,"threads_started":1}
I20260925 10:42:48.182617 7475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:48.167732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55138 (local address 127.23.100.1:43897)
0925 10:42:48.169877 (+ 2145us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:48.169888 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:48.170554 (+ 666us) server_negotiation.cc:407] Connection header received
0925 10:42:48.170659 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:48.170665 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:48.170856 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:48.171059 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:42:48.171741 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:48.173646 (+ 1905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:48.176533 (+ 2887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:48.177939 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:48.178093 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:48.178296 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:48.179818 (+ 1522us) server_negotiation.cc:299] Negotiation successful
0925 10:42:48.179996 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1952,"thread_start_us":1812,"threads_started":1}
I20260925 10:42:48.180012 7473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:48.167546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.2:39469 (local address 127.0.0.1:48536)
0925 10:42:48.171050 (+ 3504us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:48.171077 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 10:42:48.171260 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:42:48.171852 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:42:48.171856 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:42:48.171881 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:42:48.172234 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:48.172245 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:48.174233 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:48.174238 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:48.176062 (+ 1824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:48.176076 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:48.178455 (+ 2379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:48.178462 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:48.178724 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:48.179430 (+ 706us) client_negotiation.cc:769] Sending connection context
0925 10:42:48.179545 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 10:42:48.179686 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3341,"mutex_wait_us":747,"thread_start_us":2442,"threads_started":1}
I20260925 10:42:48.182685 7394 tablet_service.cc:1467] Processing CreateTablet for tablet 27300e99ed034302b3457b27dc031eaa (DEFAULT_TABLE table=test-table [id=5d0614e6d23d47058113513749c53ffb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:42:48.184135 7394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27300e99ed034302b3457b27dc031eaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:42:48.185693 7127 tablet_service.cc:1467] Processing CreateTablet for tablet 27300e99ed034302b3457b27dc031eaa (DEFAULT_TABLE table=test-table [id=5d0614e6d23d47058113513749c53ffb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:42:48.186137 7261 tablet_service.cc:1467] Processing CreateTablet for tablet 27300e99ed034302b3457b27dc031eaa (DEFAULT_TABLE table=test-table [id=5d0614e6d23d47058113513749c53ffb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:42:48.186985 7127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27300e99ed034302b3457b27dc031eaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:42:48.187422 7261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27300e99ed034302b3457b27dc031eaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:42:48.197201 7477 tablet_bootstrap.cc:492] T 27300e99ed034302b3457b27dc031eaa P 24c5c47272fe4d52b77ba1edc0d1e120: Bootstrap starting.
I20260925 10:42:48.199143 7478 tablet_bootstrap.cc:492] T 27300e99ed034302b3457b27dc031eaa P 938822573aaf469abe7f961a07d8e36b: Bootstrap starting.
I20260925 10:42:48.201115 7479 tablet_bootstrap.cc:492] T 27300e99ed034302b3457b27dc031eaa P bd045bbe55db4881bcd6d93cbcc2a3af: Bootstrap starting.
I20260925 10:42:48.201735 7477 tablet_bootstrap.cc:654] T 27300e99ed034302b3457b27dc031eaa P 24c5c47272fe4d52b77ba1edc0d1e120: Neither blocks nor log segments found. Creating new log.
I20260925 10:42:48.202538 7478 tablet_bootstrap.cc:654] T 27300e99ed034302b3457b27dc031eaa P 938822573aaf469abe7f961a07d8e36b: Neither blocks nor log segments found. Creating new log.
I20260925 10:42:48.203061 7477 log.cc:824] T 27300e99ed034302b3457b27dc031eaa P 24c5c47272fe4d52b77ba1edc0d1e120: Log is configured to *not* fsync() on all Append() calls
I20260925 10:42:48.203735 7478 log.cc:824] T 27300e99ed034302b3457b27dc031eaa P 938822573aaf469abe7f961a07d8e36b: Log is configured to *not* fsync() on all Append() calls
I20260925 10:42:48.203951 7479 tablet_bootstrap.cc:654] T 27300e99ed034302b3457b27dc031eaa P bd045bbe55db4881bcd6d93cbcc2a3af: Neither blocks nor log segments found. Creating new log.
I20260925 10:42:48.204900 7479 log.cc:824] T 27300e99ed034302b3457b27dc031eaa P bd045bbe55db4881bcd6d93cbcc2a3af: Log is configured to *not* fsync() on all Append() calls
I20260925 10:42:48.206440 7477 tablet_bootstrap.cc:492] T 27300e99ed034302b3457b27dc031eaa P 24c5c47272fe4d52b77ba1edc0d1e120: No bootstrap required, opened a new log
I20260925 10:42:48.206756 7477 ts_tablet_manager.cc:1397] T 27300e99ed034302b3457b27dc031eaa P 24c5c47272fe4d52b77ba1edc0d1e120: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260925 10:42:48.207293 7479 tablet_bootstrap.cc:492] T 27300e99ed034302b3457b27dc031eaa P bd045bbe55db4881bcd6d93cbcc2a3af: No bootstrap required, opened a new log
I20260925 10:42:48.207535 7479 ts_tablet_manager.cc:1397] T 27300e99ed034302b3457b27dc031eaa P bd045bbe55db4881bcd6d93cbcc2a3af: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 10:42:48.212666 7478 tablet_bootstrap.cc:492] T 27300e99ed034302b3457b27dc031eaa P 938822573aaf469abe7f961a07d8e36b: No bootstrap required, opened a new log
I20260925 10:42:48.212915 7478 ts_tablet_manager.cc:1397] T 27300e99ed034302b3457b27dc031eaa P 938822573aaf469abe7f961a07d8e36b: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.011s
I20260925 10:42:48.215184 7479 raft_consensus.cc:348] T 27300e99ed034302b3457b27dc031eaa P bd045bbe55db4881bcd6d93cbcc2a3af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "938822573aaf469abe7f961a07d8e36b" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 44349 } } peers { permanent_uuid: "24c5c47272fe4d52b77ba1edc0d1e120" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 43897 } } peers { permanent_uuid: "bd045bbe55db4881bcd6d93cbcc2a3af" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 39469 } }
I20260925 10:42:48.215536 7479 raft_consensus.cc:374] T 27300e99ed034302b3457b27dc031eaa P bd045bbe55db4881bcd6d93cbcc2a3af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:42:48.215664 7479 raft_consensus.cc:729] T 27300e99ed034302b3457b27dc031eaa P bd045bbe55db4881bcd6d93cbcc2a3af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd045bbe55db4881bcd6d93cbcc2a3af, State: Initialized, Role: FOLLOWER
I20260925 10:42:48.216208 7479 consensus_queue.cc:260] T 27300e99ed034302b3457b27dc031eaa P bd045bbe55db4881bcd6d93cbcc2a3af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "938822573aaf469abe7f961a07d8e36b" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 44349 } } peers { permanent_uuid: "24c5c47272fe4d52b77ba1edc0d1e120" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 43897 } } peers { permanent_uuid: "bd045bbe55db4881bcd6d93cbcc2a3af" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 39469 } }
I20260925 10:42:48.217546 7477 raft_consensus.cc:348] T 27300e99ed034302b3457b27dc031eaa P 24c5c47272fe4d52b77ba1edc0d1e120 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "938822573aaf469abe7f961a07d8e36b" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 44349 } } peers { permanent_uuid: "24c5c47272fe4d52b77ba1edc0d1e120" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 43897 } } peers { permanent_uuid: "bd045bbe55db4881bcd6d93cbcc2a3af" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 39469 } }
I20260925 10:42:48.217933 7477 raft_consensus.cc:374] T 27300e99ed034302b3457b27dc031eaa P 24c5c47272fe4d52b77ba1edc0d1e120 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:42:48.218048 7477 raft_consensus.cc:729] T 27300e99ed034302b3457b27dc031eaa P 24c5c47272fe4d52b77ba1edc0d1e120 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24c5c47272fe4d52b77ba1edc0d1e120, State: Initialized, Role: FOLLOWER
I20260925 10:42:48.218668 7477 consensus_queue.cc:260] T 27300e99ed034302b3457b27dc031eaa P 24c5c47272fe4d52b77ba1edc0d1e120 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "938822573aaf469abe7f961a07d8e36b" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 44349 } } peers { permanent_uuid: "24c5c47272fe4d52b77ba1edc0d1e120" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 43897 } } peers { permanent_uuid: "bd045bbe55db4881bcd6d93cbcc2a3af" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 39469 } }
I20260925 10:42:48.220670 7477 ts_tablet_manager.cc:1428] T 27300e99ed034302b3457b27dc031eaa P 24c5c47272fe4d52b77ba1edc0d1e120: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260925 10:42:48.221575 7326 heartbeater.cc:503] Master 127.23.100.62:35471 was elected leader, sending a full tablet report...
I20260925 10:42:48.223243 7479 ts_tablet_manager.cc:1428] T 27300e99ed034302b3457b27dc031eaa P bd045bbe55db4881bcd6d93cbcc2a3af: Time spent starting tablet: real 0.016s user 0.007s sys 0.006s
I20260925 10:42:48.223067 7478 raft_consensus.cc:348] T 27300e99ed034302b3457b27dc031eaa P 938822573aaf469abe7f961a07d8e36b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "938822573aaf469abe7f961a07d8e36b" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 44349 } } peers { permanent_uuid: "24c5c47272fe4d52b77ba1edc0d1e120" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 43897 } } peers { permanent_uuid: "bd045bbe55db4881bcd6d93cbcc2a3af" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 39469 } }
I20260925 10:42:48.223483 7478 raft_consensus.cc:374] T 27300e99ed034302b3457b27dc031eaa P 938822573aaf469abe7f961a07d8e36b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:42:48.223614 7478 raft_consensus.cc:729] T 27300e99ed034302b3457b27dc031eaa P 938822573aaf469abe7f961a07d8e36b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 938822573aaf469abe7f961a07d8e36b, State: Initialized, Role: FOLLOWER
I20260925 10:42:48.224131 7478 consensus_queue.cc:260] T 27300e99ed034302b3457b27dc031eaa P 938822573aaf469abe7f961a07d8e36b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "938822573aaf469abe7f961a07d8e36b" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 44349 } } peers { permanent_uuid: "24c5c47272fe4d52b77ba1edc0d1e120" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 43897 } } peers { permanent_uuid: "bd045bbe55db4881bcd6d93cbcc2a3af" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 39469 } }
I20260925 10:42:48.225219 7459 heartbeater.cc:503] Master 127.23.100.62:35471 was elected leader, sending a full tablet report...
I20260925 10:42:48.225777 7478 ts_tablet_manager.cc:1428] T 27300e99ed034302b3457b27dc031eaa P 938822573aaf469abe7f961a07d8e36b: Time spent starting tablet: real 0.013s user 0.008s sys 0.006s
W20260925 10:42:48.335692 7327 tablet.cc:2369] T 27300e99ed034302b3457b27dc031eaa P bd045bbe55db4881bcd6d93cbcc2a3af: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:42:48.347615 7193 tablet.cc:2369] T 27300e99ed034302b3457b27dc031eaa P 24c5c47272fe4d52b77ba1edc0d1e120: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:42:48.349996 7460 tablet.cc:2369] T 27300e99ed034302b3457b27dc031eaa P 938822573aaf469abe7f961a07d8e36b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:42:48.396106 7484 raft_consensus.cc:482] T 27300e99ed034302b3457b27dc031eaa P 24c5c47272fe4d52b77ba1edc0d1e120 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:42:48.396445 7484 raft_consensus.cc:504] T 27300e99ed034302b3457b27dc031eaa P 24c5c47272fe4d52b77ba1edc0d1e120 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "938822573aaf469abe7f961a07d8e36b" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 44349 } } peers { permanent_uuid: "24c5c47272fe4d52b77ba1edc0d1e120" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 43897 } } peers { permanent_uuid: "bd045bbe55db4881bcd6d93cbcc2a3af" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 39469 } }
I20260925 10:42:48.398113 7484 leader_election.cc:288] T 27300e99ed034302b3457b27dc031eaa P 24c5c47272fe4d52b77ba1edc0d1e120 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 938822573aaf469abe7f961a07d8e36b (127.23.100.3:44349), bd045bbe55db4881bcd6d93cbcc2a3af (127.23.100.2:39469)
I20260925 10:42:48.405310 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:48.398535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:48801 (local address 127.23.100.3:44349)
0925 10:42:48.398763 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:48.398771 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:48.399564 (+ 793us) server_negotiation.cc:407] Connection header received
0925 10:42:48.399730 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:48.399736 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:48.399810 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:48.399913 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:42:48.400504 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:48.401277 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:48.403366 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:48.404341 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:48.404532 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:48.404610 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:48.404921 (+ 311us) server_negotiation.cc:299] Negotiation successful
0925 10:42:48.405051 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260925 10:42:48.405362 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:48.398775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:54655 (local address 127.23.100.2:39469)
0925 10:42:48.399029 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:48.399038 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:48.399730 (+ 692us) server_negotiation.cc:407] Connection header received
0925 10:42:48.399909 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:48.399914 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:48.400000 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:48.400102 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:42:48.400687 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:48.401389 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:48.403418 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:48.404380 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:48.404557 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:48.404636 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:48.404983 (+ 347us) server_negotiation.cc:299] Negotiation successful
0925 10:42:48.405107 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 10:42:48.405453 7486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:48.398535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.3:44349 (local address 127.23.100.1:48801)
0925 10:42:48.399439 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:48.399469 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 10:42:48.399593 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:42:48.399938 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:42:48.399942 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:42:48.399965 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:42:48.400356 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:48.400366 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:48.401422 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:48.401430 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:48.403223 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:48.403234 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:48.404572 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:48.404578 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:48.404772 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:48.404794 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 10:42:48.404895 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 10:42:48.405014 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"mutex_wait_us":20,"thread_start_us":453,"threads_started":1}
I20260925 10:42:48.405498 7487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:48.398607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.2:39469 (local address 127.23.100.1:54655)
0925 10:42:48.399604 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0925 10:42:48.399625 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 10:42:48.399770 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:42:48.400126 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:42:48.400129 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:42:48.400151 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:42:48.400520 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:48.400530 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:48.401529 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:48.401536 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:48.403229 (+ 1693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:42:48.403242 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:42:48.404621 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:42:48.404627 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:42:48.404823 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:48.404846 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 10:42:48.404959 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 10:42:48.405075 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":854,"mutex_wait_us":79,"thread_start_us":602,"threads_started":1}
I20260925 10:42:48.406631 7414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27300e99ed034302b3457b27dc031eaa" candidate_uuid: "24c5c47272fe4d52b77ba1edc0d1e120" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "938822573aaf469abe7f961a07d8e36b" is_pre_election: true
I20260925 10:42:48.406634 7281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27300e99ed034302b3457b27dc031eaa" candidate_uuid: "24c5c47272fe4d52b77ba1edc0d1e120" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd045bbe55db4881bcd6d93cbcc2a3af" is_pre_election: true
I20260925 10:42:48.407122 7281 raft_consensus.cc:2454] T 27300e99ed034302b3457b27dc031eaa P bd045bbe55db4881bcd6d93cbcc2a3af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24c5c47272fe4d52b77ba1edc0d1e120 in term 0.
I20260925 10:42:48.407121 7414 raft_consensus.cc:2454] T 27300e99ed034302b3457b27dc031eaa P 938822573aaf469abe7f961a07d8e36b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24c5c47272fe4d52b77ba1edc0d1e120 in term 0.
I20260925 10:42:48.407735 7082 leader_election.cc:302] T 27300e99ed034302b3457b27dc031eaa P 24c5c47272fe4d52b77ba1edc0d1e120 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24c5c47272fe4d52b77ba1edc0d1e120, 938822573aaf469abe7f961a07d8e36b; no voters:
I20260925 10:42:48.408128 7484 raft_consensus.cc:2781] T 27300e99ed034302b3457b27dc031eaa P 24c5c47272fe4d52b77ba1edc0d1e120 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:42:48.408272 7484 raft_consensus.cc:482] T 27300e99ed034302b3457b27dc031eaa P 24c5c47272fe4d52b77ba1edc0d1e120 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:42:48.408407 7484 raft_consensus.cc:3037] T 27300e99ed034302b3457b27dc031eaa P 24c5c47272fe4d52b77ba1edc0d1e120 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:42:48.410743 7484 raft_consensus.cc:504] T 27300e99ed034302b3457b27dc031eaa P 24c5c47272fe4d52b77ba1edc0d1e120 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "938822573aaf469abe7f961a07d8e36b" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 44349 } } peers { permanent_uuid: "24c5c47272fe4d52b77ba1edc0d1e120" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 43897 } } peers { permanent_uuid: "bd045bbe55db4881bcd6d93cbcc2a3af" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 39469 } }
I20260925 10:42:48.411309 7484 leader_election.cc:288] T 27300e99ed034302b3457b27dc031eaa P 24c5c47272fe4d52b77ba1edc0d1e120 [CANDIDATE]: Term 1 election: Requested vote from peers 938822573aaf469abe7f961a07d8e36b (127.23.100.3:44349), bd045bbe55db4881bcd6d93cbcc2a3af (127.23.100.2:39469)
I20260925 10:42:48.411749 7414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27300e99ed034302b3457b27dc031eaa" candidate_uuid: "24c5c47272fe4d52b77ba1edc0d1e120" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "938822573aaf469abe7f961a07d8e36b"
I20260925 10:42:48.411748 7281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27300e99ed034302b3457b27dc031eaa" candidate_uuid: "24c5c47272fe4d52b77ba1edc0d1e120" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd045bbe55db4881bcd6d93cbcc2a3af"
I20260925 10:42:48.411937 7414 raft_consensus.cc:3037] T 27300e99ed034302b3457b27dc031eaa P 938822573aaf469abe7f961a07d8e36b [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:42:48.411937 7281 raft_consensus.cc:3037] T 27300e99ed034302b3457b27dc031eaa P bd045bbe55db4881bcd6d93cbcc2a3af [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:42:48.414992 7414 raft_consensus.cc:2454] T 27300e99ed034302b3457b27dc031eaa P 938822573aaf469abe7f961a07d8e36b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24c5c47272fe4d52b77ba1edc0d1e120 in term 1.
I20260925 10:42:48.415001 7281 raft_consensus.cc:2454] T 27300e99ed034302b3457b27dc031eaa P bd045bbe55db4881bcd6d93cbcc2a3af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24c5c47272fe4d52b77ba1edc0d1e120 in term 1.
I20260925 10:42:48.415361 7081 leader_election.cc:302] T 27300e99ed034302b3457b27dc031eaa P 24c5c47272fe4d52b77ba1edc0d1e120 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24c5c47272fe4d52b77ba1edc0d1e120, bd045bbe55db4881bcd6d93cbcc2a3af; no voters:
I20260925 10:42:48.415663 7484 raft_consensus.cc:2781] T 27300e99ed034302b3457b27dc031eaa P 24c5c47272fe4d52b77ba1edc0d1e120 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:42:48.416121 7484 raft_consensus.cc:686] T 27300e99ed034302b3457b27dc031eaa P 24c5c47272fe4d52b77ba1edc0d1e120 [term 1 LEADER]: Becoming Leader. State: Replica: 24c5c47272fe4d52b77ba1edc0d1e120, State: Running, Role: LEADER
I20260925 10:42:48.416551 7484 consensus_queue.cc:237] T 27300e99ed034302b3457b27dc031eaa P 24c5c47272fe4d52b77ba1edc0d1e120 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "938822573aaf469abe7f961a07d8e36b" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 44349 } } peers { permanent_uuid: "24c5c47272fe4d52b77ba1edc0d1e120" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 43897 } } peers { permanent_uuid: "bd045bbe55db4881bcd6d93cbcc2a3af" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 39469 } }
I20260925 10:42:48.420718 7002 catalog_manager.cc:5495] T 27300e99ed034302b3457b27dc031eaa P 24c5c47272fe4d52b77ba1edc0d1e120 reported cstate change: term changed from 0 to 1, leader changed from <none> to 24c5c47272fe4d52b77ba1edc0d1e120 (127.23.100.1). New cstate: current_term: 1 leader_uuid: "24c5c47272fe4d52b77ba1edc0d1e120" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "938822573aaf469abe7f961a07d8e36b" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 44349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24c5c47272fe4d52b77ba1edc0d1e120" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 43897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd045bbe55db4881bcd6d93cbcc2a3af" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 39469 } health_report { overall_health: UNKNOWN } } }
I20260925 10:42:48.472194 7475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:48.467223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55142 (local address 127.23.100.1:43897)
0925 10:42:48.467422 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:48.467426 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:48.467496 (+ 70us) server_negotiation.cc:407] Connection header received
0925 10:42:48.467676 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:48.467680 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:48.467757 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:48.467886 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:42:48.468188 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:48.468903 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:48.470282 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:48.470694 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:48.470778 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:48.471269 (+ 491us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:42:48.471738 (+ 469us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:42:48.471851 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:48.471940 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 10:42:48.472009 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 10:42:48.480110 7281 raft_consensus.cc:1261] T 27300e99ed034302b3457b27dc031eaa P bd045bbe55db4881bcd6d93cbcc2a3af [term 1 FOLLOWER]: Refusing update from remote peer 24c5c47272fe4d52b77ba1edc0d1e120: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:42:48.480110 7414 raft_consensus.cc:1261] T 27300e99ed034302b3457b27dc031eaa P 938822573aaf469abe7f961a07d8e36b [term 1 FOLLOWER]: Refusing update from remote peer 24c5c47272fe4d52b77ba1edc0d1e120: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:42:48.481030 7484 consensus_queue.cc:1035] T 27300e99ed034302b3457b27dc031eaa P 24c5c47272fe4d52b77ba1edc0d1e120 [LEADER]: Connected to new peer: Peer: permanent_uuid: "938822573aaf469abe7f961a07d8e36b" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 44349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:42:48.481309 7488 consensus_queue.cc:1035] T 27300e99ed034302b3457b27dc031eaa P 24c5c47272fe4d52b77ba1edc0d1e120 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd045bbe55db4881bcd6d93cbcc2a3af" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 39469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:42:48.491112 7491 mvcc.cc:205] Tried to move back new op lower bound from 7333203838884716544 to 7333203838637789184. Current Snapshot: MvccSnapshot[applied={T|T < 7333203838884716544}]
I20260925 10:42:48.494597 7492 mvcc.cc:205] Tried to move back new op lower bound from 7333203838884716544 to 7333203838637789184. Current Snapshot: MvccSnapshot[applied={T|T < 7333203838884716544}]
I20260925 10:42:48.506560 7003 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:42:48.506839 7003 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:42:48.510213 7003 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d: Sending DeleteTablet for 3 replicas of tablet 27300e99ed034302b3457b27dc031eaa
I20260925 10:42:48.511266 7394 tablet_service.cc:1514] Processing DeleteTablet for tablet 27300e99ed034302b3457b27dc031eaa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:42:48 UTC) from {username='kudu', principal='kudu/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:50610
I20260925 10:42:48.511308 7127 tablet_service.cc:1514] Processing DeleteTablet for tablet 27300e99ed034302b3457b27dc031eaa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:42:48 UTC) from {username='kudu', principal='kudu/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:55138
I20260925 10:42:48.511759 7261 tablet_service.cc:1514] Processing DeleteTablet for tablet 27300e99ed034302b3457b27dc031eaa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:42:48 UTC) from {username='kudu', principal='kudu/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:48536
I20260925 10:42:48.512668 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 7064
I20260925 10:42:48.535655 7501 tablet_replica.cc:330] T 27300e99ed034302b3457b27dc031eaa P 24c5c47272fe4d52b77ba1edc0d1e120: stopping tablet replica
I20260925 10:42:48.536391 7501 raft_consensus.cc:2229] T 27300e99ed034302b3457b27dc031eaa P 24c5c47272fe4d52b77ba1edc0d1e120 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:42:48.538771 7501 raft_consensus.cc:2258] T 27300e99ed034302b3457b27dc031eaa P 24c5c47272fe4d52b77ba1edc0d1e120 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:42:48.538846 7502 tablet_replica.cc:330] T 27300e99ed034302b3457b27dc031eaa P bd045bbe55db4881bcd6d93cbcc2a3af: stopping tablet replica
I20260925 10:42:48.539664 7500 tablet_replica.cc:330] T 27300e99ed034302b3457b27dc031eaa P 938822573aaf469abe7f961a07d8e36b: stopping tablet replica
I20260925 10:42:48.540122 7500 raft_consensus.cc:2229] T 27300e99ed034302b3457b27dc031eaa P 938822573aaf469abe7f961a07d8e36b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:42:48.540715 7500 raft_consensus.cc:2258] T 27300e99ed034302b3457b27dc031eaa P 938822573aaf469abe7f961a07d8e36b [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 25 10:42:48 dist-test-slave-dxdd krb5kdc[6958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.1@KRBTEST.COM
I20260925 10:42:48.542747 7500 ts_tablet_manager.cc:1905] T 27300e99ed034302b3457b27dc031eaa P 938822573aaf469abe7f961a07d8e36b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:42:48.544351 7501 ts_tablet_manager.cc:1905] T 27300e99ed034302b3457b27dc031eaa P 24c5c47272fe4d52b77ba1edc0d1e120: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:42:48.550732 7502 raft_consensus.cc:2229] T 27300e99ed034302b3457b27dc031eaa P bd045bbe55db4881bcd6d93cbcc2a3af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:42:48.551242 7502 raft_consensus.cc:2258] T 27300e99ed034302b3457b27dc031eaa P bd045bbe55db4881bcd6d93cbcc2a3af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:42:48.553004 7501 ts_tablet_manager.cc:1918] T 27300e99ed034302b3457b27dc031eaa P 24c5c47272fe4d52b77ba1edc0d1e120: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:42:48.553205 7501 log.cc:1196] T 27300e99ed034302b3457b27dc031eaa P 24c5c47272fe4d52b77ba1edc0d1e120: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-0/wal/wals/27300e99ed034302b3457b27dc031eaa
I20260925 10:42:48.553262 7502 ts_tablet_manager.cc:1905] T 27300e99ed034302b3457b27dc031eaa P bd045bbe55db4881bcd6d93cbcc2a3af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:42:48.553732 7501 ts_tablet_manager.cc:1939] T 27300e99ed034302b3457b27dc031eaa P 24c5c47272fe4d52b77ba1edc0d1e120: Deleting consensus metadata
I20260925 10:42:48.555212 7475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:48.513565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55156 (local address 127.23.100.1:43897)
0925 10:42:48.513746 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:48.513754 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:48.522866 (+ 9112us) server_negotiation.cc:407] Connection header received
0925 10:42:48.523906 (+ 1040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:48.523914 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:48.524023 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:48.524112 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:42:48.529215 (+ 5103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:48.530746 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:48.536844 (+ 6098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:48.537787 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:48.537910 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:48.543430 (+ 5520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:42:48.543444 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:42:48.543462 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:42:48.543511 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:42:48.547785 (+ 4274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:48.548605 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:48.548613 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:48.548617 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:48.548813 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:48.549360 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:48.549366 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:48.549369 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:48.549836 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:42:48.550156 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:48.554576 (+ 4420us) server_negotiation.cc:299] Negotiation successful
0925 10:42:48.554844 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 10:42:48.555579 6987 catalog_manager.cc:4841] TS 24c5c47272fe4d52b77ba1edc0d1e120 (127.23.100.1:43897): tablet 27300e99ed034302b3457b27dc031eaa (table test-table [id=5d0614e6d23d47058113513749c53ffb]) successfully deleted
I20260925 10:42:48.556274 7187 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:42:48.558406 7500 ts_tablet_manager.cc:1918] T 27300e99ed034302b3457b27dc031eaa P 938822573aaf469abe7f961a07d8e36b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:42:48.558633 7500 log.cc:1196] T 27300e99ed034302b3457b27dc031eaa P 938822573aaf469abe7f961a07d8e36b: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-2/wal/wals/27300e99ed034302b3457b27dc031eaa
I20260925 10:42:48.559239 7500 ts_tablet_manager.cc:1939] T 27300e99ed034302b3457b27dc031eaa P 938822573aaf469abe7f961a07d8e36b: Deleting consensus metadata
I20260925 10:42:48.561017 6989 catalog_manager.cc:4841] TS 938822573aaf469abe7f961a07d8e36b (127.23.100.3:44349): tablet 27300e99ed034302b3457b27dc031eaa (table test-table [id=5d0614e6d23d47058113513749c53ffb]) successfully deleted
I20260925 10:42:48.561848 7502 ts_tablet_manager.cc:1918] T 27300e99ed034302b3457b27dc031eaa P bd045bbe55db4881bcd6d93cbcc2a3af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:42:48.562011 7502 log.cc:1196] T 27300e99ed034302b3457b27dc031eaa P bd045bbe55db4881bcd6d93cbcc2a3af: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-1/wal/wals/27300e99ed034302b3457b27dc031eaa
I20260925 10:42:48.562505 7502 ts_tablet_manager.cc:1939] T 27300e99ed034302b3457b27dc031eaa P bd045bbe55db4881bcd6d93cbcc2a3af: Deleting consensus metadata
I20260925 10:42:48.563879 6988 catalog_manager.cc:4841] TS bd045bbe55db4881bcd6d93cbcc2a3af (127.23.100.2:39469): tablet 27300e99ed034302b3457b27dc031eaa (table test-table [id=5d0614e6d23d47058113513749c53ffb]) successfully deleted
I20260925 10:42:48.654067 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 7064
I20260925 10:42:48.671692 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 7198
Sep 25 10:42:48 dist-test-slave-dxdd krb5kdc[6958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.2@KRBTEST.COM
I20260925 10:42:48.686153 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:48.672536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48546 (local address 127.23.100.2:39469)
0925 10:42:48.672707 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:48.672711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:48.672728 (+ 17us) server_negotiation.cc:407] Connection header received
0925 10:42:48.672884 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:48.672887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:48.672954 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:48.673039 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:42:48.674370 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:48.675259 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:48.676638 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:48.677019 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:48.677105 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:48.681189 (+ 4084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:42:48.681201 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:42:48.681216 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:42:48.681248 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:42:48.683921 (+ 2673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:48.684475 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:48.684481 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:48.684483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:48.684597 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:48.684895 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:48.684898 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:48.684900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:48.685237 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:42:48.685441 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:48.685656 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 10:42:48.685798 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 10:42:48.686893 7321 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:42:48.775463 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 7198
I20260925 10:42:48.792311 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 7331
Sep 25 10:42:48 dist-test-slave-dxdd krb5kdc[6958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.3@KRBTEST.COM
I20260925 10:42:48.806906 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:42:48.793121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50624 (local address 127.23.100.3:44349)
0925 10:42:48.793295 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 10:42:48.793299 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:42:48.793316 (+ 17us) server_negotiation.cc:407] Connection header received
0925 10:42:48.793372 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:42:48.793376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:42:48.793446 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:42:48.793534 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:42:48.794919 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:48.795631 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:48.796877 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:42:48.797243 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:42:48.797332 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:42:48.801655 (+ 4323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:42:48.801670 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:42:48.801686 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:42:48.801715 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:42:48.804170 (+ 2455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:48.804790 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:48.804796 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:48.804799 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:48.804917 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:42:48.805213 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:42:48.805216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:42:48.805218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:42:48.805531 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:42:48.805726 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0925 10:42:48.805949 (+ 223us) server_negotiation.cc:299] Negotiation successful
0925 10:42:48.806630 (+ 681us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 10:42:48.807569 7454 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:42:48.894769 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 7331
I20260925 10:42:48.911173 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 6971
I20260925 10:42:48.912249 7033 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:42:49.023762 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 6971
I20260925 10:43:09.045128 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:35471
--webserver_interface=127.23.100.62
--webserver_port=46277
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:38443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.62:35471
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:43:09.364895 7508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:43:09.365187 7508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:43:09.365334 7508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:43:09.372536 7508 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 10:43:09.372655 7508 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 10:43:09.372746 7508 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 10:43:09.372839 7508 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:43:09.372941 7508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:43:09.373044 7508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:43:09.373114 7508 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:43:09.373196 7508 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:43:09.382526 7508 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:38443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.23.100.62:35471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:35471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.62
--webserver_port=46277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:43:09.383693 7508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:43:09.384989 7508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:43:09.406944 7517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:43:09.406915 7514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:43:09.407012 7515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:43:09.407784 7508 server_base.cc:1034] running on GCE node
I20260925 10:43:09.408601 7508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:43:09.410053 7508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:43:09.411264 7508 hybrid_clock.cc:648] HybridClock initialized: now 1790332989411172 us; error 63 us; skew 500 ppm
I20260925 10:43:09.411600 7508 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:43:09 dist-test-slave-dxdd krb5kdc[6958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332989, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:43:09.415028 7508 init.cc:377] Logged in from keytab as kudu/127.23.100.62@KRBTEST.COM (short username kudu)
I20260925 10:43:09.416492 7508 webserver.cc:460] Webserver started at http://127.23.100.62:46277/ using document root <none> and password file <none>
I20260925 10:43:09.417212 7508 fs_manager.cc:359] Metadata directory not provided
I20260925 10:43:09.417309 7508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:43:09.422976 7508 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 10:43:09.425743 7524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:43:09.427138 7508 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 10:43:09.427312 7508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/master-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/master-0/wal
uuid: "282881ffd90e4e42a826159c2fe1678d"
format_stamp: "Formatted at 2026-09-25 10:42:46 on dist-test-slave-dxdd"
server_key: "6d6e6f0211224999f567176237198484"
server_key_iv: "04f228acc47f06b9b946e5ed5ad5c5ee"
server_key_version: "encryptionkey@0"
I20260925 10:43:09.428357 7508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:43:09.455646 7508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:43:09.459525 7508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:43:09.459825 7508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:43:09.478018 7508 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.62:35471
I20260925 10:43:09.478048 7575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.62:35471 every 8 connection(s)
I20260925 10:43:09.479532 7508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
I20260925 10:43:09.480301 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 7508
I20260925 10:43:09.481231 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.1:43897
--local_ip_for_outbound_sockets=127.23.100.1
--tserver_master_addrs=127.23.100.62:35471
--webserver_port=40453
--webserver_interface=127.23.100.1
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab
--principal=kudu/127.23.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:38443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:43:09.486555 7576 sys_catalog.cc:263] Verifying existing consensus state
I20260925 10:43:09.489706 7576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d: Bootstrap starting.
I20260925 10:43:09.508713 7576 log.cc:824] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d: Log is configured to *not* fsync() on all Append() calls
I20260925 10:43:09.520067 7576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:43:09.520561 7576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d: Bootstrap complete.
I20260925 10:43:09.529408 7576 raft_consensus.cc:348] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "282881ffd90e4e42a826159c2fe1678d" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 35471 } }
I20260925 10:43:09.530342 7576 raft_consensus.cc:729] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 282881ffd90e4e42a826159c2fe1678d, State: Initialized, Role: FOLLOWER
I20260925 10:43:09.531021 7576 consensus_queue.cc:260] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "282881ffd90e4e42a826159c2fe1678d" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 35471 } }
I20260925 10:43:09.531349 7576 raft_consensus.cc:388] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:43:09.531484 7576 raft_consensus.cc:482] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:43:09.531667 7576 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d [term 1 FOLLOWER]: Advancing to term 2
I20260925 10:43:09.535678 7576 raft_consensus.cc:504] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "282881ffd90e4e42a826159c2fe1678d" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 35471 } }
I20260925 10:43:09.536176 7576 leader_election.cc:302] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 282881ffd90e4e42a826159c2fe1678d; no voters:
I20260925 10:43:09.536773 7576 leader_election.cc:288] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 10:43:09.536978 7580 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d [term 2 FOLLOWER]: Leader election won for term 2
I20260925 10:43:09.538111 7580 raft_consensus.cc:686] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d [term 2 LEADER]: Becoming Leader. State: Replica: 282881ffd90e4e42a826159c2fe1678d, State: Running, Role: LEADER
I20260925 10:43:09.538883 7576 sys_catalog.cc:564] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:43:09.538793 7580 consensus_queue.cc:237] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "282881ffd90e4e42a826159c2fe1678d" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 35471 } }
I20260925 10:43:09.547086 7582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 282881ffd90e4e42a826159c2fe1678d. Latest consensus state: current_term: 2 leader_uuid: "282881ffd90e4e42a826159c2fe1678d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "282881ffd90e4e42a826159c2fe1678d" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 35471 } } }
I20260925 10:43:09.547461 7582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d [sys.catalog]: This master's current role is: LEADER
I20260925 10:43:09.548056 7581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "282881ffd90e4e42a826159c2fe1678d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "282881ffd90e4e42a826159c2fe1678d" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 35471 } } }
I20260925 10:43:09.548424 7581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d [sys.catalog]: This master's current role is: LEADER
I20260925 10:43:09.561622 7594 catalog_manager.cc:1259] Loaded cluster ID: 78d8f9c6914141d38315fd6f65ef128f
I20260925 10:43:09.561718 7594 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d: loading cluster ID for follower catalog manager: success
I20260925 10:43:09.565521 7594 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d: acquiring CA information for follower catalog manager: success
I20260925 10:43:09.568003 7594 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 10:43:09.568200 7596 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:43:09.572501 7596 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:43:09.573647 7596 catalog_manager.cc:1259] Loaded cluster ID: 78d8f9c6914141d38315fd6f65ef128f
I20260925 10:43:09.573738 7596 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:43:09.575423 7596 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:43:09.576573 7596 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d: Loaded TSK: 0
I20260925 10:43:09.610430 7596 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d: Generated new TSK 1
I20260925 10:43:09.610826 7596 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 10:43:09.814221 7578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:43:09.814607 7578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:43:09.814772 7578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:43:09.821741 7578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:43:09.821879 7578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:43:09.822067 7578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.1
I20260925 10:43:09.831213 7578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:38443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab
--principal=kudu/127.23.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.1:43897
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.1
--webserver_port=40453
--tserver_master_addrs=127.23.100.62:35471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.1
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:43:09.832505 7578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:43:09.833776 7578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:43:09.846259 7603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:43:09.846930 7604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:43:09.847081 7606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:43:09.847464 7578 server_base.cc:1034] running on GCE node
I20260925 10:43:09.848052 7578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:43:09.848939 7578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:43:09.850188 7578 hybrid_clock.cc:648] HybridClock initialized: now 1790332989850121 us; error 51 us; skew 500 ppm
I20260925 10:43:09.850557 7578 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:43:09 dist-test-slave-dxdd krb5kdc[6958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332989, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:43:09.853691 7578 init.cc:377] Logged in from keytab as kudu/127.23.100.1@KRBTEST.COM (short username kudu)
I20260925 10:43:09.855264 7578 webserver.cc:460] Webserver started at http://127.23.100.1:40453/ using document root <none> and password file <none>
I20260925 10:43:09.855902 7578 fs_manager.cc:359] Metadata directory not provided
I20260925 10:43:09.856066 7578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:43:09.861668 7578 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 10:43:09.864506 7613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:43:09.865875 7578 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:43:09.866032 7578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-0/wal
uuid: "24c5c47272fe4d52b77ba1edc0d1e120"
format_stamp: "Formatted at 2026-09-25 10:42:47 on dist-test-slave-dxdd"
server_key: "aa9594b95baee257d2bd937db21816b5"
server_key_iv: "c678bb0bf00dafa9173c703922bdfd3c"
server_key_version: "encryptionkey@0"
I20260925 10:43:09.867117 7578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:43:09.884822 7578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:43:09.889051 7578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:43:09.889318 7578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:43:09.890381 7578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:43:09.891667 7578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:43:09.891757 7578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:43:09.891902 7578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:43:09.892004 7578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:43:09.926617 7578 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.1:43897
I20260925 10:43:09.926678 7725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.1:43897 every 8 connection(s)
I20260925 10:43:09.928011 7578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
I20260925 10:43:09.933601 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 7578
I20260925 10:43:09.934438 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.2:39469
--local_ip_for_outbound_sockets=127.23.100.2
--tserver_master_addrs=127.23.100.62:35471
--webserver_port=42107
--webserver_interface=127.23.100.2
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab
--principal=kudu/127.23.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:38443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 10:43:09 dist-test-slave-dxdd krb5kdc[6958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332989, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.1@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:43:09.954381 7729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:09.931014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.1:36807)
0925 10:43:09.931765 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:09.931803 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 10:43:09.934077 (+ 2274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:43:09.935526 (+ 1449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:43:09.935540 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:43:09.936311 (+ 771us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:43:09.937063 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:09.937089 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:09.938864 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:09.938872 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:09.941121 (+ 2249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:09.941139 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:09.942295 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:09.942322 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:09.942640 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:09.943321 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:43:09.943344 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:43:09.945692 (+ 2348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:43:09.950234 (+ 4542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:43:09.950240 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:43:09.950251 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:43:09.950727 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:43:09.951220 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:43:09.951223 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:43:09.951226 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:43:09.951401 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:43:09.952326 (+ 925us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:43:09.952331 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:43:09.952626 (+ 295us) client_negotiation.cc:769] Sending connection context
0925 10:43:09.952837 (+ 211us) client_negotiation.cc:240] Negotiation successful
0925 10:43:09.953293 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":287,"threads_started":1}
I20260925 10:43:09.954860 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:09.930812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:36807 (local address 127.23.100.62:35471)
0925 10:43:09.931499 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0925 10:43:09.931515 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 10:43:09.931980 (+ 465us) server_negotiation.cc:407] Connection header received
0925 10:43:09.934696 (+ 2716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:43:09.934714 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:43:09.935027 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:43:09.935292 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:43:09.937260 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:09.938686 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:09.941263 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:09.942057 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:09.942183 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:09.945966 (+ 3783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:43:09.945978 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:43:09.945997 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:43:09.946050 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:43:09.950079 (+ 4029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:43:09.950862 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:43:09.950872 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:43:09.950883 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:43:09.951084 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:43:09.951523 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:43:09.951531 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:43:09.951537 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:43:09.952192 (+ 655us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:43:09.952397 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 10:43:09.953143 (+ 746us) server_negotiation.cc:299] Negotiation successful
0925 10:43:09.953577 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":220,"threads_started":1}
I20260925 10:43:09.957579 7726 heartbeater.cc:347] Connected to a master server at 127.23.100.62:35471
I20260925 10:43:09.957875 7726 heartbeater.cc:464] Registering TS with master...
I20260925 10:43:09.958657 7726 heartbeater.cc:511] Master 127.23.100.62:35471 requested a full tablet report, sending...
I20260925 10:43:09.960866 7541 ts_manager.cc:194] Registered new tserver with Master: 24c5c47272fe4d52b77ba1edc0d1e120 (127.23.100.1:43897)
I20260925 10:43:09.962393 7541 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.1@KRBTEST.COM'} at 127.23.100.1:36807
W20260925 10:43:10.257345 7730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:43:10.257642 7730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:43:10.257802 7730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:43:10.264652 7730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:43:10.264815 7730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:43:10.264979 7730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.2
I20260925 10:43:10.274464 7730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:38443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab
--principal=kudu/127.23.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.2:39469
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.2
--webserver_port=42107
--tserver_master_addrs=127.23.100.62:35471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.2
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:43:10.275648 7730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:43:10.276908 7730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:43:10.289247 7735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:43:10.289196 7736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:43:10.289284 7738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:43:10.290071 7730 server_base.cc:1034] running on GCE node
I20260925 10:43:10.290717 7730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:43:10.291589 7730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:43:10.292814 7730 hybrid_clock.cc:648] HybridClock initialized: now 1790332990292778 us; error 47 us; skew 500 ppm
I20260925 10:43:10.293112 7730 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:43:10 dist-test-slave-dxdd krb5kdc[6958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332990, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:43:10.296133 7730 init.cc:377] Logged in from keytab as kudu/127.23.100.2@KRBTEST.COM (short username kudu)
I20260925 10:43:10.297516 7730 webserver.cc:460] Webserver started at http://127.23.100.2:42107/ using document root <none> and password file <none>
I20260925 10:43:10.298110 7730 fs_manager.cc:359] Metadata directory not provided
I20260925 10:43:10.298226 7730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:43:10.303186 7730 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 10:43:10.305593 7745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:43:10.306950 7730 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:43:10.307116 7730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-1/wal
uuid: "bd045bbe55db4881bcd6d93cbcc2a3af"
format_stamp: "Formatted at 2026-09-25 10:42:47 on dist-test-slave-dxdd"
server_key: "b0535cac90591377fca569b392e5d358"
server_key_iv: "b08fc578239d0f300681f7df6f7ad797"
server_key_version: "encryptionkey@0"
I20260925 10:43:10.308010 7730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:43:10.354492 7730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:43:10.358407 7730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:43:10.358633 7730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:43:10.359531 7730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:43:10.360754 7730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:43:10.360836 7730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:43:10.360908 7730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:43:10.360958 7730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:43:10.394635 7730 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.2:39469
I20260925 10:43:10.394671 7857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.2:39469 every 8 connection(s)
I20260925 10:43:10.396035 7730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
I20260925 10:43:10.401633 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 7730
I20260925 10:43:10.402406 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.3:44349
--local_ip_for_outbound_sockets=127.23.100.3
--tserver_master_addrs=127.23.100.62:35471
--webserver_port=40667
--webserver_interface=127.23.100.3
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab
--principal=kudu/127.23.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:38443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 10:43:10 dist-test-slave-dxdd krb5kdc[6958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332990, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.2@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:43:10.419484 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:10.398484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:55003 (local address 127.23.100.62:35471)
0925 10:43:10.398656 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 10:43:10.398661 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:43:10.399848 (+ 1187us) server_negotiation.cc:407] Connection header received
0925 10:43:10.402509 (+ 2661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:43:10.402515 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:43:10.402598 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:43:10.402711 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:43:10.404738 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:10.405438 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:10.407949 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:10.408360 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:10.408454 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:10.412641 (+ 4187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:43:10.412649 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:43:10.412657 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:43:10.412703 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:43:10.416225 (+ 3522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:43:10.416926 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:43:10.416933 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:43:10.416940 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:43:10.417087 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:43:10.417489 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:43:10.417497 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:43:10.417503 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:43:10.417896 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:43:10.418040 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 10:43:10.418746 (+ 706us) server_negotiation.cc:299] Negotiation successful
0925 10:43:10.419002 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 10:43:10.420058 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:10.398955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.2:55003)
0925 10:43:10.399662 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:10.399711 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 10:43:10.402131 (+ 2420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:43:10.402978 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:43:10.402993 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:43:10.403785 (+ 792us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:43:10.404561 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:10.404587 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:10.405595 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:10.405602 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:10.407796 (+ 2194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:10.407813 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:10.408577 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:10.408586 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:10.408913 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:10.409694 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:43:10.409736 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:43:10.412423 (+ 2687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:43:10.416372 (+ 3949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:43:10.416378 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:43:10.416389 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:43:10.416799 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:43:10.417188 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:43:10.417191 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:43:10.417193 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:43:10.417375 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:43:10.418012 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:43:10.418018 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:43:10.418367 (+ 349us) client_negotiation.cc:769] Sending connection context
0925 10:43:10.418598 (+ 231us) client_negotiation.cc:240] Negotiation successful
0925 10:43:10.418961 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":268,"threads_started":1}
I20260925 10:43:10.421492 7858 heartbeater.cc:347] Connected to a master server at 127.23.100.62:35471
I20260925 10:43:10.421800 7858 heartbeater.cc:464] Registering TS with master...
I20260925 10:43:10.422636 7858 heartbeater.cc:511] Master 127.23.100.62:35471 requested a full tablet report, sending...
I20260925 10:43:10.424204 7541 ts_manager.cc:194] Registered new tserver with Master: bd045bbe55db4881bcd6d93cbcc2a3af (127.23.100.2:39469)
I20260925 10:43:10.425499 7541 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.2@KRBTEST.COM'} at 127.23.100.2:55003
W20260925 10:43:10.725661 7861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:43:10.725916 7861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:43:10.726006 7861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:43:10.733019 7861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:43:10.733114 7861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:43:10.733225 7861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.3
I20260925 10:43:10.742479 7861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:38443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab
--principal=kudu/127.23.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.3:44349
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.3
--webserver_port=40667
--tserver_master_addrs=127.23.100.62:35471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.3
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:43:10.743549 7861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:43:10.744813 7861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:43:10.757313 7866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:43:10.757377 7869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:43:10.757402 7867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:43:10.758128 7861 server_base.cc:1034] running on GCE node
I20260925 10:43:10.758757 7861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:43:10.759616 7861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:43:10.760841 7861 hybrid_clock.cc:648] HybridClock initialized: now 1790332990760803 us; error 58 us; skew 500 ppm
I20260925 10:43:10.761134 7861 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:43:10 dist-test-slave-dxdd krb5kdc[6958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332990, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:43:10.767958 7861 init.cc:377] Logged in from keytab as kudu/127.23.100.3@KRBTEST.COM (short username kudu)
I20260925 10:43:10.769526 7861 webserver.cc:460] Webserver started at http://127.23.100.3:40667/ using document root <none> and password file <none>
I20260925 10:43:10.770206 7861 fs_manager.cc:359] Metadata directory not provided
I20260925 10:43:10.770396 7861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:43:10.775424 7861 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260925 10:43:10.777940 7876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:43:10.779197 7861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 10:43:10.779392 7861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-2/wal
uuid: "938822573aaf469abe7f961a07d8e36b"
format_stamp: "Formatted at 2026-09-25 10:42:48 on dist-test-slave-dxdd"
server_key: "c8114d8716ce625d6069bd7f1d2f73e3"
server_key_iv: "99b959de31f0f6579b68096f16f12e02"
server_key_version: "encryptionkey@0"
I20260925 10:43:10.780357 7861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:43:10.793651 7861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:43:10.797403 7861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:43:10.797646 7861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:43:10.798645 7861 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:43:10.799881 7861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:43:10.800001 7861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:43:10.800114 7861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:43:10.800195 7861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:43:10.832095 7861 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.3:44349
I20260925 10:43:10.832242 7988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.3:44349 every 8 connection(s)
I20260925 10:43:10.833513 7861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
I20260925 10:43:10.839900 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 7861
Sep 25 10:43:10 dist-test-slave-dxdd krb5kdc[6958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332990, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.3@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:43:10.855288 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:10.835958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:54553 (local address 127.23.100.62:35471)
0925 10:43:10.836159 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 10:43:10.836165 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:43:10.837371 (+ 1206us) server_negotiation.cc:407] Connection header received
0925 10:43:10.839343 (+ 1972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:43:10.839347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:43:10.839409 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:43:10.839495 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:43:10.841546 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:10.842570 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:10.844767 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:10.845426 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:10.845507 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:10.849290 (+ 3783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:43:10.849294 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:43:10.849298 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:43:10.849325 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:43:10.852322 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:43:10.853010 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:43:10.853017 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:43:10.853020 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:43:10.853139 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:43:10.853565 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:43:10.853572 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:43:10.853577 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:43:10.853839 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:43:10.853984 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 10:43:10.854713 (+ 729us) server_negotiation.cc:299] Negotiation successful
0925 10:43:10.854942 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 10:43:10.855754 7992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:10.840879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49662 (local address 127.23.100.62:35471)
0925 10:43:10.841442 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0925 10:43:10.841448 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:43:10.841794 (+ 346us) server_negotiation.cc:407] Connection header received
0925 10:43:10.841927 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:43:10.841932 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:43:10.842003 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:43:10.842105 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:43:10.844295 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:10.845214 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:10.847090 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:10.847490 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:10.847578 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:10.850367 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:43:10.850374 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:43:10.850379 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:43:10.850432 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:43:10.853038 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:43:10.853693 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:43:10.853701 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:43:10.853704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:43:10.853826 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:43:10.854188 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:43:10.854194 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:43:10.854197 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:43:10.854539 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:43:10.854809 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0925 10:43:10.855109 (+ 300us) server_negotiation.cc:299] Negotiation successful
0925 10:43:10.855335 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":277,"threads_started":1}
I20260925 10:43:10.856504 7991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:10.836481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.3:54553)
0925 10:43:10.837197 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:10.837245 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 10:43:10.838986 (+ 1741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:43:10.839784 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:43:10.839795 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:43:10.840562 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:43:10.841384 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:10.841411 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:10.842765 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:10.842773 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:10.844632 (+ 1859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:10.844647 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:10.845642 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:10.845649 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:10.845919 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:10.846648 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:43:10.846681 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:43:10.849073 (+ 2392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:43:10.852459 (+ 3386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:43:10.852469 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:43:10.852485 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:43:10.852894 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:43:10.853267 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:43:10.853271 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:43:10.853275 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:43:10.853455 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:43:10.853972 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:43:10.853980 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:43:10.854297 (+ 317us) client_negotiation.cc:769] Sending connection context
0925 10:43:10.854619 (+ 322us) client_negotiation.cc:240] Negotiation successful
0925 10:43:10.855125 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":312,"threads_started":1}
I20260925 10:43:10.857861 7989 heartbeater.cc:347] Connected to a master server at 127.23.100.62:35471
I20260925 10:43:10.858197 7989 heartbeater.cc:464] Registering TS with master...
I20260925 10:43:10.858928 7989 heartbeater.cc:511] Master 127.23.100.62:35471 requested a full tablet report, sending...
I20260925 10:43:10.860209 7540 ts_manager.cc:194] Registered new tserver with Master: 938822573aaf469abe7f961a07d8e36b (127.23.100.3:44349)
I20260925 10:43:10.861205 7540 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.3@KRBTEST.COM'} at 127.23.100.3:54553
I20260925 10:43:10.870761 23952 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:43:10.876796 7992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:10.871661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49678 (local address 127.23.100.62:35471)
0925 10:43:10.871809 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 10:43:10.871813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:43:10.872306 (+ 493us) server_negotiation.cc:407] Connection header received
0925 10:43:10.872469 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:43:10.872472 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:43:10.872588 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:43:10.872705 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:43:10.873103 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:10.873771 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:10.875257 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:10.875658 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:10.875749 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:10.876189 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:43:10.876381 (+ 192us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 10:43:10.876619 (+ 238us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49678: token expired
Metrics: {"server-negotiator.queue_time_us":55}
W20260925 10:43:10.876924 7992 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49678: token expired
W20260925 10:43:10.877447 7465 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.23.100.62:35471)
I20260925 10:43:10.889971 7992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:10.877887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49686 (local address 127.23.100.62:35471)
0925 10:43:10.878048 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 10:43:10.878055 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:43:10.878089 (+ 34us) server_negotiation.cc:407] Connection header received
0925 10:43:10.878245 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:43:10.878250 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:43:10.878354 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:43:10.878456 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:43:10.879787 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:10.880454 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:10.881825 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:10.882218 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:10.882375 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:10.885049 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:43:10.885054 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:43:10.885059 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:43:10.885096 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:43:10.887461 (+ 2365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:43:10.888102 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:43:10.888109 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:43:10.888112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:43:10.888246 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:43:10.888627 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:43:10.888633 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:43:10.888636 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:43:10.888913 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:43:10.889156 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0925 10:43:10.889430 (+ 274us) server_negotiation.cc:299] Negotiation successful
0925 10:43:10.889633 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 10:43:10.892970 7540 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49686:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:43:10.899941 7540 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:43:10.939028 8000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:10.910676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36836 (local address 127.23.100.1:43897)
0925 10:43:10.930395 (+ 19719us) server_negotiation.cc:206] Beginning negotiation
0925 10:43:10.930404 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:43:10.930432 (+ 28us) server_negotiation.cc:407] Connection header received
0925 10:43:10.930520 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:43:10.930527 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:43:10.930760 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:43:10.930946 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:43:10.932067 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:10.933747 (+ 1680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:10.936512 (+ 2765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:10.937655 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:10.937831 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:10.938064 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0925 10:43:10.938489 (+ 425us) server_negotiation.cc:299] Negotiation successful
0925 10:43:10.938652 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19544,"thread_start_us":19289,"threads_started":1}
I20260925 10:43:10.939165 7999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:10.910467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:43897 (local address 127.0.0.1:36836)
0925 10:43:10.926812 (+ 16345us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:10.926858 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 10:43:10.927063 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:43:10.931250 (+ 4187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:43:10.931258 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:43:10.931294 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:43:10.931855 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:10.931873 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:10.933940 (+ 2067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:10.933945 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:10.936325 (+ 2380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:10.936342 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:10.937910 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:10.937916 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:10.938184 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:10.938220 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 10:43:10.938438 (+ 218us) client_negotiation.cc:240] Negotiation successful
0925 10:43:10.938661 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15447,"mutex_wait_us":44,"thread_start_us":15294,"threads_started":1}
I20260925 10:43:10.941942 8001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:10.911135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.3:44349 (local address 127.0.0.1:43742)
0925 10:43:10.933167 (+ 22032us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:10.933189 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 10:43:10.933322 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:43:10.934075 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:43:10.934081 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:43:10.934114 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:43:10.934526 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:10.934536 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:10.935905 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:10.935909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:10.938850 (+ 2941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:10.938864 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:10.941029 (+ 2165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:10.941034 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:10.941282 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:10.941351 (+ 69us) client_negotiation.cc:769] Sending connection context
0925 10:43:10.941433 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 10:43:10.941578 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21921,"mutex_wait_us":548,"thread_start_us":21268,"threads_started":1}
I20260925 10:43:10.942025 8003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:10.911737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38004 (local address 127.23.100.2:39469)
0925 10:43:10.932219 (+ 20482us) server_negotiation.cc:206] Beginning negotiation
0925 10:43:10.932227 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:43:10.932931 (+ 704us) server_negotiation.cc:407] Connection header received
0925 10:43:10.933022 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:43:10.933029 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:43:10.933266 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:43:10.933423 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:43:10.934176 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:10.935664 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:10.939074 (+ 3410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:10.940508 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:10.940634 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:10.940813 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 10:43:10.941544 (+ 731us) server_negotiation.cc:299] Negotiation successful
0925 10:43:10.941716 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20314,"thread_start_us":20216,"threads_started":1}
I20260925 10:43:10.942029 8004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:10.912182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43742 (local address 127.23.100.3:44349)
0925 10:43:10.931820 (+ 19638us) server_negotiation.cc:206] Beginning negotiation
0925 10:43:10.931831 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:43:10.933498 (+ 1667us) server_negotiation.cc:407] Connection header received
0925 10:43:10.933604 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:43:10.933612 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:43:10.933837 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:43:10.934043 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:43:10.934710 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:10.935705 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:10.939074 (+ 3369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:10.940508 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:10.940634 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:10.940847 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0925 10:43:10.941521 (+ 674us) server_negotiation.cc:299] Negotiation successful
0925 10:43:10.941670 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19435,"thread_start_us":19262,"threads_started":1}
I20260925 10:43:10.942665 8002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:10.910658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.2:39469 (local address 127.0.0.1:38004)
0925 10:43:10.932669 (+ 22011us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:10.932689 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:43:10.932808 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:43:10.933590 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:43:10.933594 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:43:10.933616 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:43:10.933952 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:10.933964 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:10.935813 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:10.935820 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:10.938850 (+ 3030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:10.938864 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:10.941018 (+ 2154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:10.941024 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:10.941232 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:10.941349 (+ 117us) client_negotiation.cc:769] Sending connection context
0925 10:43:10.941436 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 10:43:10.942292 (+ 856us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21881,"mutex_wait_us":826,"thread_start_us":22240,"threads_started":1}
I20260925 10:43:10.943329 7661 tablet_service.cc:1467] Processing CreateTablet for tablet f955ef6fb4024db885e3795d39f1c85f (DEFAULT_TABLE table=test-table [id=3eec72b12a454bb480606f537ae654d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:43:10.944854 7661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f955ef6fb4024db885e3795d39f1c85f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:43:10.945489 7924 tablet_service.cc:1467] Processing CreateTablet for tablet f955ef6fb4024db885e3795d39f1c85f (DEFAULT_TABLE table=test-table [id=3eec72b12a454bb480606f537ae654d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:43:10.946060 7793 tablet_service.cc:1467] Processing CreateTablet for tablet f955ef6fb4024db885e3795d39f1c85f (DEFAULT_TABLE table=test-table [id=3eec72b12a454bb480606f537ae654d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:43:10.946913 7924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f955ef6fb4024db885e3795d39f1c85f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:43:10.947337 7793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f955ef6fb4024db885e3795d39f1c85f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:43:10.959758 8005 tablet_bootstrap.cc:492] T f955ef6fb4024db885e3795d39f1c85f P 938822573aaf469abe7f961a07d8e36b: Bootstrap starting.
I20260925 10:43:10.961748 8007 tablet_bootstrap.cc:492] T f955ef6fb4024db885e3795d39f1c85f P bd045bbe55db4881bcd6d93cbcc2a3af: Bootstrap starting.
I20260925 10:43:10.963649 8005 tablet_bootstrap.cc:654] T f955ef6fb4024db885e3795d39f1c85f P 938822573aaf469abe7f961a07d8e36b: Neither blocks nor log segments found. Creating new log.
I20260925 10:43:10.964906 8005 log.cc:824] T f955ef6fb4024db885e3795d39f1c85f P 938822573aaf469abe7f961a07d8e36b: Log is configured to *not* fsync() on all Append() calls
I20260925 10:43:10.965306 8007 tablet_bootstrap.cc:654] T f955ef6fb4024db885e3795d39f1c85f P bd045bbe55db4881bcd6d93cbcc2a3af: Neither blocks nor log segments found. Creating new log.
I20260925 10:43:10.965488 7726 heartbeater.cc:503] Master 127.23.100.62:35471 was elected leader, sending a full tablet report...
I20260925 10:43:10.966549 8007 log.cc:824] T f955ef6fb4024db885e3795d39f1c85f P bd045bbe55db4881bcd6d93cbcc2a3af: Log is configured to *not* fsync() on all Append() calls
I20260925 10:43:10.966284 8006 tablet_bootstrap.cc:492] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120: Bootstrap starting.
I20260925 10:43:10.970793 8006 tablet_bootstrap.cc:654] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120: Neither blocks nor log segments found. Creating new log.
I20260925 10:43:10.971840 8006 log.cc:824] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120: Log is configured to *not* fsync() on all Append() calls
I20260925 10:43:10.974671 8006 tablet_bootstrap.cc:492] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120: No bootstrap required, opened a new log
I20260925 10:43:10.974922 8006 ts_tablet_manager.cc:1397] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20260925 10:43:10.975196 8007 tablet_bootstrap.cc:492] T f955ef6fb4024db885e3795d39f1c85f P bd045bbe55db4881bcd6d93cbcc2a3af: No bootstrap required, opened a new log
I20260925 10:43:10.975445 8007 ts_tablet_manager.cc:1397] T f955ef6fb4024db885e3795d39f1c85f P bd045bbe55db4881bcd6d93cbcc2a3af: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20260925 10:43:10.975869 8005 tablet_bootstrap.cc:492] T f955ef6fb4024db885e3795d39f1c85f P 938822573aaf469abe7f961a07d8e36b: No bootstrap required, opened a new log
I20260925 10:43:10.976121 8005 ts_tablet_manager.cc:1397] T f955ef6fb4024db885e3795d39f1c85f P 938822573aaf469abe7f961a07d8e36b: Time spent bootstrapping tablet: real 0.017s user 0.013s sys 0.000s
I20260925 10:43:10.981745 8006 raft_consensus.cc:348] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c5c47272fe4d52b77ba1edc0d1e120" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 43897 } } peers { permanent_uuid: "bd045bbe55db4881bcd6d93cbcc2a3af" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 39469 } } peers { permanent_uuid: "938822573aaf469abe7f961a07d8e36b" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 44349 } }
I20260925 10:43:10.982096 8006 raft_consensus.cc:374] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:43:10.982214 8006 raft_consensus.cc:729] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24c5c47272fe4d52b77ba1edc0d1e120, State: Initialized, Role: FOLLOWER
I20260925 10:43:10.982796 8006 consensus_queue.cc:260] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c5c47272fe4d52b77ba1edc0d1e120" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 43897 } } peers { permanent_uuid: "bd045bbe55db4881bcd6d93cbcc2a3af" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 39469 } } peers { permanent_uuid: "938822573aaf469abe7f961a07d8e36b" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 44349 } }
I20260925 10:43:10.984520 8006 ts_tablet_manager.cc:1428] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120: Time spent starting tablet: real 0.009s user 0.010s sys 0.000s
I20260925 10:43:10.985754 8007 raft_consensus.cc:348] T f955ef6fb4024db885e3795d39f1c85f P bd045bbe55db4881bcd6d93cbcc2a3af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c5c47272fe4d52b77ba1edc0d1e120" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 43897 } } peers { permanent_uuid: "bd045bbe55db4881bcd6d93cbcc2a3af" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 39469 } } peers { permanent_uuid: "938822573aaf469abe7f961a07d8e36b" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 44349 } }
I20260925 10:43:10.986100 8007 raft_consensus.cc:374] T f955ef6fb4024db885e3795d39f1c85f P bd045bbe55db4881bcd6d93cbcc2a3af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:43:10.986217 8007 raft_consensus.cc:729] T f955ef6fb4024db885e3795d39f1c85f P bd045bbe55db4881bcd6d93cbcc2a3af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd045bbe55db4881bcd6d93cbcc2a3af, State: Initialized, Role: FOLLOWER
I20260925 10:43:10.986846 8007 consensus_queue.cc:260] T f955ef6fb4024db885e3795d39f1c85f P bd045bbe55db4881bcd6d93cbcc2a3af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c5c47272fe4d52b77ba1edc0d1e120" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 43897 } } peers { permanent_uuid: "bd045bbe55db4881bcd6d93cbcc2a3af" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 39469 } } peers { permanent_uuid: "938822573aaf469abe7f961a07d8e36b" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 44349 } }
I20260925 10:43:10.986899 8005 raft_consensus.cc:348] T f955ef6fb4024db885e3795d39f1c85f P 938822573aaf469abe7f961a07d8e36b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c5c47272fe4d52b77ba1edc0d1e120" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 43897 } } peers { permanent_uuid: "bd045bbe55db4881bcd6d93cbcc2a3af" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 39469 } } peers { permanent_uuid: "938822573aaf469abe7f961a07d8e36b" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 44349 } }
I20260925 10:43:10.987215 8005 raft_consensus.cc:374] T f955ef6fb4024db885e3795d39f1c85f P 938822573aaf469abe7f961a07d8e36b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:43:10.987308 8005 raft_consensus.cc:729] T f955ef6fb4024db885e3795d39f1c85f P 938822573aaf469abe7f961a07d8e36b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 938822573aaf469abe7f961a07d8e36b, State: Initialized, Role: FOLLOWER
I20260925 10:43:10.987913 7858 heartbeater.cc:503] Master 127.23.100.62:35471 was elected leader, sending a full tablet report...
I20260925 10:43:10.987924 8005 consensus_queue.cc:260] T f955ef6fb4024db885e3795d39f1c85f P 938822573aaf469abe7f961a07d8e36b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c5c47272fe4d52b77ba1edc0d1e120" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 43897 } } peers { permanent_uuid: "bd045bbe55db4881bcd6d93cbcc2a3af" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 39469 } } peers { permanent_uuid: "938822573aaf469abe7f961a07d8e36b" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 44349 } }
I20260925 10:43:10.988577 8007 ts_tablet_manager.cc:1428] T f955ef6fb4024db885e3795d39f1c85f P bd045bbe55db4881bcd6d93cbcc2a3af: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260925 10:43:10.989507 7989 heartbeater.cc:503] Master 127.23.100.62:35471 was elected leader, sending a full tablet report...
I20260925 10:43:10.990108 8005 ts_tablet_manager.cc:1428] T f955ef6fb4024db885e3795d39f1c85f P 938822573aaf469abe7f961a07d8e36b: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
W20260925 10:43:11.085116 7990 tablet.cc:2369] T f955ef6fb4024db885e3795d39f1c85f P 938822573aaf469abe7f961a07d8e36b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:43:11.148016 7859 tablet.cc:2369] T f955ef6fb4024db885e3795d39f1c85f P bd045bbe55db4881bcd6d93cbcc2a3af: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:43:11.180352 7727 tablet.cc:2369] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:43:11.189937 8011 raft_consensus.cc:482] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:43:11.190332 8011 raft_consensus.cc:504] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c5c47272fe4d52b77ba1edc0d1e120" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 43897 } } peers { permanent_uuid: "bd045bbe55db4881bcd6d93cbcc2a3af" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 39469 } } peers { permanent_uuid: "938822573aaf469abe7f961a07d8e36b" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 44349 } }
I20260925 10:43:11.192747 8011 leader_election.cc:288] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd045bbe55db4881bcd6d93cbcc2a3af (127.23.100.2:39469), 938822573aaf469abe7f961a07d8e36b (127.23.100.3:44349)
I20260925 10:43:11.201007 8003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:11.193397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:32773 (local address 127.23.100.2:39469)
0925 10:43:11.193638 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0925 10:43:11.193643 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:43:11.194168 (+ 525us) server_negotiation.cc:407] Connection header received
0925 10:43:11.194526 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:43:11.194532 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:43:11.194610 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:43:11.194696 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:43:11.195470 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:11.196824 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:11.199281 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:11.200026 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:11.200203 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:11.200294 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 10:43:11.200583 (+ 289us) server_negotiation.cc:299] Negotiation successful
0925 10:43:11.200737 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 10:43:11.201169 8014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:11.192964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.2:39469 (local address 127.23.100.1:32773)
0925 10:43:11.194058 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:11.194079 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 10:43:11.194325 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:43:11.194740 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:43:11.194746 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:43:11.194769 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:43:11.195301 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:11.195319 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:11.196970 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:11.196975 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:11.199080 (+ 2105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:11.199090 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:11.200243 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:11.200249 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:11.200413 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:11.200452 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 10:43:11.200549 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 10:43:11.200672 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":917,"mutex_wait_us":49,"thread_start_us":686,"threads_started":1}
I20260925 10:43:11.202253 8004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:11.193003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:36057 (local address 127.23.100.3:44349)
0925 10:43:11.193218 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 10:43:11.193227 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:43:11.194567 (+ 1340us) server_negotiation.cc:407] Connection header received
0925 10:43:11.194734 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:43:11.194741 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:43:11.194828 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:43:11.194923 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:43:11.195591 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:11.196505 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:11.199105 (+ 2600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:11.200780 (+ 1675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:11.200965 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:11.201060 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 10:43:11.201856 (+ 796us) server_negotiation.cc:299] Negotiation successful
0925 10:43:11.202007 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 10:43:11.202193 7813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f955ef6fb4024db885e3795d39f1c85f" candidate_uuid: "24c5c47272fe4d52b77ba1edc0d1e120" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd045bbe55db4881bcd6d93cbcc2a3af" is_pre_election: true
I20260925 10:43:11.202608 8015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:11.193614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.3:44349 (local address 127.23.100.1:36057)
0925 10:43:11.194433 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:11.194450 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:43:11.194589 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:43:11.194955 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:43:11.194961 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:43:11.194991 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:43:11.195410 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:11.195422 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:11.196632 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:11.196639 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:11.198927 (+ 2288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:11.198947 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:11.201422 (+ 2475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:11.201431 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:11.201666 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:11.201724 (+ 58us) client_negotiation.cc:769] Sending connection context
0925 10:43:11.201824 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 10:43:11.202000 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"mutex_wait_us":2,"thread_start_us":478,"threads_started":1}
I20260925 10:43:11.202802 7813 raft_consensus.cc:2454] T f955ef6fb4024db885e3795d39f1c85f P bd045bbe55db4881bcd6d93cbcc2a3af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24c5c47272fe4d52b77ba1edc0d1e120 in term 0.
I20260925 10:43:11.203589 7615 leader_election.cc:302] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24c5c47272fe4d52b77ba1edc0d1e120, bd045bbe55db4881bcd6d93cbcc2a3af; no voters:
I20260925 10:43:11.203644 7944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f955ef6fb4024db885e3795d39f1c85f" candidate_uuid: "24c5c47272fe4d52b77ba1edc0d1e120" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "938822573aaf469abe7f961a07d8e36b" is_pre_election: true
I20260925 10:43:11.204079 8011 raft_consensus.cc:2781] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:43:11.204145 7944 raft_consensus.cc:2454] T f955ef6fb4024db885e3795d39f1c85f P 938822573aaf469abe7f961a07d8e36b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24c5c47272fe4d52b77ba1edc0d1e120 in term 0.
I20260925 10:43:11.204265 8011 raft_consensus.cc:482] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:43:11.204380 8011 raft_consensus.cc:3037] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:43:11.208287 8011 raft_consensus.cc:504] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c5c47272fe4d52b77ba1edc0d1e120" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 43897 } } peers { permanent_uuid: "bd045bbe55db4881bcd6d93cbcc2a3af" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 39469 } } peers { permanent_uuid: "938822573aaf469abe7f961a07d8e36b" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 44349 } }
I20260925 10:43:11.209105 8011 leader_election.cc:288] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120 [CANDIDATE]: Term 1 election: Requested vote from peers bd045bbe55db4881bcd6d93cbcc2a3af (127.23.100.2:39469), 938822573aaf469abe7f961a07d8e36b (127.23.100.3:44349)
I20260925 10:43:11.209582 7944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f955ef6fb4024db885e3795d39f1c85f" candidate_uuid: "24c5c47272fe4d52b77ba1edc0d1e120" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "938822573aaf469abe7f961a07d8e36b"
I20260925 10:43:11.209550 7813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f955ef6fb4024db885e3795d39f1c85f" candidate_uuid: "24c5c47272fe4d52b77ba1edc0d1e120" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd045bbe55db4881bcd6d93cbcc2a3af"
I20260925 10:43:11.209789 7944 raft_consensus.cc:3037] T f955ef6fb4024db885e3795d39f1c85f P 938822573aaf469abe7f961a07d8e36b [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:43:11.209777 7813 raft_consensus.cc:3037] T f955ef6fb4024db885e3795d39f1c85f P bd045bbe55db4881bcd6d93cbcc2a3af [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:43:11.212138 7944 raft_consensus.cc:2454] T f955ef6fb4024db885e3795d39f1c85f P 938822573aaf469abe7f961a07d8e36b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24c5c47272fe4d52b77ba1edc0d1e120 in term 1.
I20260925 10:43:11.212633 7616 leader_election.cc:302] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24c5c47272fe4d52b77ba1edc0d1e120, 938822573aaf469abe7f961a07d8e36b; no voters:
I20260925 10:43:11.212950 8011 raft_consensus.cc:2781] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:43:11.213534 8011 raft_consensus.cc:686] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120 [term 1 LEADER]: Becoming Leader. State: Replica: 24c5c47272fe4d52b77ba1edc0d1e120, State: Running, Role: LEADER
I20260925 10:43:11.213577 7813 raft_consensus.cc:2454] T f955ef6fb4024db885e3795d39f1c85f P bd045bbe55db4881bcd6d93cbcc2a3af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24c5c47272fe4d52b77ba1edc0d1e120 in term 1.
I20260925 10:43:11.214169 8011 consensus_queue.cc:237] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c5c47272fe4d52b77ba1edc0d1e120" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 43897 } } peers { permanent_uuid: "bd045bbe55db4881bcd6d93cbcc2a3af" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 39469 } } peers { permanent_uuid: "938822573aaf469abe7f961a07d8e36b" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 44349 } }
I20260925 10:43:11.218497 7540 catalog_manager.cc:5495] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120 reported cstate change: term changed from 0 to 1, leader changed from <none> to 24c5c47272fe4d52b77ba1edc0d1e120 (127.23.100.1). New cstate: current_term: 1 leader_uuid: "24c5c47272fe4d52b77ba1edc0d1e120" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c5c47272fe4d52b77ba1edc0d1e120" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 43897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd045bbe55db4881bcd6d93cbcc2a3af" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 39469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "938822573aaf469abe7f961a07d8e36b" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 44349 } health_report { overall_health: UNKNOWN } } }
I20260925 10:43:11.241843 8000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:11.235051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36848 (local address 127.23.100.1:43897)
0925 10:43:11.235284 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 10:43:11.235292 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:43:11.235316 (+ 24us) server_negotiation.cc:407] Connection header received
0925 10:43:11.235423 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:43:11.235430 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:43:11.235547 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:43:11.235736 (+ 189us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:43:11.236066 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:11.237416 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:11.238835 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:11.239761 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:11.239904 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:11.240381 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:43:11.241160 (+ 779us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:43:11.241344 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 10:43:11.241401 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 10:43:11.241495 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 10:43:11.251400 7944 raft_consensus.cc:1261] T f955ef6fb4024db885e3795d39f1c85f P 938822573aaf469abe7f961a07d8e36b [term 1 FOLLOWER]: Refusing update from remote peer 24c5c47272fe4d52b77ba1edc0d1e120: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:43:11.251397 7813 raft_consensus.cc:1261] T f955ef6fb4024db885e3795d39f1c85f P bd045bbe55db4881bcd6d93cbcc2a3af [term 1 FOLLOWER]: Refusing update from remote peer 24c5c47272fe4d52b77ba1edc0d1e120: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:43:11.252341 8011 consensus_queue.cc:1035] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd045bbe55db4881bcd6d93cbcc2a3af" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 39469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:43:11.252627 8016 consensus_queue.cc:1035] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120 [LEADER]: Connected to new peer: Peer: permanent_uuid: "938822573aaf469abe7f961a07d8e36b" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 44349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:43:11.269917 8020 mvcc.cc:205] Tried to move back new op lower bound from 7333203932155953152 to 7333203932017766400. Current Snapshot: MvccSnapshot[applied={T|T < 7333203932155953152}]
I20260925 10:43:11.286273 7540 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:43:11.286610 7540 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:43:11.291152 7540 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d: Sending DeleteTablet for 3 replicas of tablet f955ef6fb4024db885e3795d39f1c85f
I20260925 10:43:11.292310 7661 tablet_service.cc:1514] Processing DeleteTablet for tablet f955ef6fb4024db885e3795d39f1c85f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:43:11 UTC) from {username='kudu', principal='kudu/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:36836
I20260925 10:43:11.293212 7924 tablet_service.cc:1514] Processing DeleteTablet for tablet f955ef6fb4024db885e3795d39f1c85f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:43:11 UTC) from {username='kudu', principal='kudu/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:43742
I20260925 10:43:11.293830 7793 tablet_service.cc:1514] Processing DeleteTablet for tablet f955ef6fb4024db885e3795d39f1c85f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:43:11 UTC) from {username='kudu', principal='kudu/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:38004
I20260925 10:43:11.295650 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 7578
I20260925 10:43:11.331559 8000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:11.297149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36856 (local address 127.23.100.1:43897)
0925 10:43:11.297342 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 10:43:11.297348 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:43:11.297365 (+ 17us) server_negotiation.cc:407] Connection header received
0925 10:43:11.297424 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:43:11.297430 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:43:11.297502 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:43:11.297601 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:43:11.299659 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:11.300885 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:11.304904 (+ 4019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:11.305639 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:11.305760 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:11.319531 (+ 13771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:43:11.319546 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:43:11.319565 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:43:11.319612 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:43:11.323662 (+ 4050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:43:11.324671 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:43:11.324681 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:43:11.324687 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:43:11.324866 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:43:11.326800 (+ 1934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:43:11.326807 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:43:11.326812 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:43:11.327279 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:43:11.327581 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0925 10:43:11.329592 (+ 2011us) server_negotiation.cc:299] Negotiation successful
0925 10:43:11.329847 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 10:43:11.332326 7721 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:43:11.344002 8030 tablet_replica.cc:330] T f955ef6fb4024db885e3795d39f1c85f P bd045bbe55db4881bcd6d93cbcc2a3af: stopping tablet replica
I20260925 10:43:11.344607 8030 raft_consensus.cc:2229] T f955ef6fb4024db885e3795d39f1c85f P bd045bbe55db4881bcd6d93cbcc2a3af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:43:11.345255 8030 raft_consensus.cc:2258] T f955ef6fb4024db885e3795d39f1c85f P bd045bbe55db4881bcd6d93cbcc2a3af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:43:11.346254 8029 tablet_replica.cc:330] T f955ef6fb4024db885e3795d39f1c85f P 938822573aaf469abe7f961a07d8e36b: stopping tablet replica
I20260925 10:43:11.346740 8029 raft_consensus.cc:2229] T f955ef6fb4024db885e3795d39f1c85f P 938822573aaf469abe7f961a07d8e36b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:43:11.347230 8029 raft_consensus.cc:2258] T f955ef6fb4024db885e3795d39f1c85f P 938822573aaf469abe7f961a07d8e36b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:43:11.348969 8030 ts_tablet_manager.cc:1905] T f955ef6fb4024db885e3795d39f1c85f P bd045bbe55db4881bcd6d93cbcc2a3af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:43:11.349025 8029 ts_tablet_manager.cc:1905] T f955ef6fb4024db885e3795d39f1c85f P 938822573aaf469abe7f961a07d8e36b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:43:11.355816 8030 ts_tablet_manager.cc:1918] T f955ef6fb4024db885e3795d39f1c85f P bd045bbe55db4881bcd6d93cbcc2a3af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:43:11.355816 8029 ts_tablet_manager.cc:1918] T f955ef6fb4024db885e3795d39f1c85f P 938822573aaf469abe7f961a07d8e36b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:43:11.355964 8029 log.cc:1196] T f955ef6fb4024db885e3795d39f1c85f P 938822573aaf469abe7f961a07d8e36b: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-2/wal/wals/f955ef6fb4024db885e3795d39f1c85f
I20260925 10:43:11.356060 8030 log.cc:1196] T f955ef6fb4024db885e3795d39f1c85f P bd045bbe55db4881bcd6d93cbcc2a3af: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-1/wal/wals/f955ef6fb4024db885e3795d39f1c85f
I20260925 10:43:11.356556 8029 ts_tablet_manager.cc:1939] T f955ef6fb4024db885e3795d39f1c85f P 938822573aaf469abe7f961a07d8e36b: Deleting consensus metadata
I20260925 10:43:11.356559 8030 ts_tablet_manager.cc:1939] T f955ef6fb4024db885e3795d39f1c85f P bd045bbe55db4881bcd6d93cbcc2a3af: Deleting consensus metadata
I20260925 10:43:11.358402 7526 catalog_manager.cc:4841] TS bd045bbe55db4881bcd6d93cbcc2a3af (127.23.100.2:39469): tablet f955ef6fb4024db885e3795d39f1c85f (table test-table [id=3eec72b12a454bb480606f537ae654d5]) successfully deleted
I20260925 10:43:11.358383 7527 catalog_manager.cc:4841] TS 938822573aaf469abe7f961a07d8e36b (127.23.100.3:44349): tablet f955ef6fb4024db885e3795d39f1c85f (table test-table [id=3eec72b12a454bb480606f537ae654d5]) successfully deleted
I20260925 10:43:11.442569 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 7578
I20260925 10:43:11.462138 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 7730
W20260925 10:43:11.462118 7525 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.100.1:43897 (error 108)
I20260925 10:43:11.464064 8033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:11.463350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:43897 (local address 127.0.0.1:60124)
0925 10:43:11.463785 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:11.463894 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.1:43897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":236,"threads_started":1}
W20260925 10:43:11.464448 7525 catalog_manager.cc:4584] TS 24c5c47272fe4d52b77ba1edc0d1e120 (127.23.100.1:43897): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f955ef6fb4024db885e3795d39f1c85f: Network error: Client connection negotiation failed: client connection to 127.23.100.1:43897: connect: Connection refused (error 111)
I20260925 10:43:11.476238 8003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:11.463124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39948 (local address 127.23.100.2:39469)
0925 10:43:11.463287 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 10:43:11.463291 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:43:11.463309 (+ 18us) server_negotiation.cc:407] Connection header received
0925 10:43:11.463413 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:43:11.463417 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:43:11.463499 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:43:11.463624 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:43:11.465197 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:11.465938 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:11.467372 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:11.467771 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:11.467907 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:11.470589 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:43:11.470602 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:43:11.470615 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:43:11.470648 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:43:11.473327 (+ 2679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:43:11.474047 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:43:11.474055 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:43:11.474057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:43:11.474183 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:43:11.474613 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:43:11.474620 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:43:11.474627 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:43:11.475063 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:43:11.475378 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0925 10:43:11.475689 (+ 311us) server_negotiation.cc:299] Negotiation successful
0925 10:43:11.475913 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 10:43:11.476881 7853 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:43:11.518039 8033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:11.517619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:43897 (local address 127.0.0.1:60134)
0925 10:43:11.517817 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:11.517911 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.1:43897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260925 10:43:11.519110 8033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:11.518781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:43897 (local address 127.0.0.1:60148)
0925 10:43:11.518925 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:11.518990 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.1:43897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 10:43:11.565624 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 7730
I20260925 10:43:11.579885 8033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:11.579320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:43897 (local address 127.0.0.1:60162)
0925 10:43:11.579569 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:11.579686 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.1:43897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260925 10:43:11.581105 8033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:11.580605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:43897 (local address 127.0.0.1:60166)
0925 10:43:11.580828 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:11.580920 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.1:43897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 10:43:11.582800 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 7861
I20260925 10:43:11.596330 8004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:11.583803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52874 (local address 127.23.100.3:44349)
0925 10:43:11.583985 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 10:43:11.583989 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:43:11.584007 (+ 18us) server_negotiation.cc:407] Connection header received
0925 10:43:11.584183 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:43:11.584188 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:43:11.584266 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:43:11.584364 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:43:11.585954 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:11.586765 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:11.587984 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:11.588360 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:11.588454 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:11.590993 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:43:11.591008 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:43:11.591025 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:43:11.591056 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:43:11.593913 (+ 2857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:43:11.594603 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:43:11.594610 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:43:11.594612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:43:11.594762 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:43:11.595064 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:43:11.595067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:43:11.595069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:43:11.595393 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:43:11.595594 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 10:43:11.595851 (+ 257us) server_negotiation.cc:299] Negotiation successful
0925 10:43:11.596006 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 10:43:11.597155 7984 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:43:11.661839 8033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:11.661311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:43897 (local address 127.0.0.1:60180)
0925 10:43:11.661547 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:11.661653 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.1:43897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260925 10:43:11.663023 8033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:11.662575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:43897 (local address 127.0.0.1:60186)
0925 10:43:11.662783 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:11.662876 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.1:43897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260925 10:43:11.684883 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 7861
I20260925 10:43:11.701339 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 7508
I20260925 10:43:11.702477 7571 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:43:11.816917 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 7508
I20260925 10:43:31.838291 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:35471
--webserver_interface=127.23.100.62
--webserver_port=46277
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:38443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.62:35471
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:43:32.157941 8037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:43:32.158274 8037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:43:32.158454 8037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:43:32.165594 8037 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 10:43:32.165730 8037 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 10:43:32.165833 8037 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 10:43:32.165930 8037 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:43:32.166033 8037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:43:32.166157 8037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:43:32.166244 8037 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:43:32.166342 8037 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:43:32.176156 8037 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:38443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.23.100.62:35471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:35471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.62
--webserver_port=46277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:43:32.177454 8037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:43:32.178864 8037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:43:32.190879 8043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:43:32.190975 8042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:43:32.190999 8045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:43:32.191599 8037 server_base.cc:1034] running on GCE node
I20260925 10:43:32.192363 8037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:43:32.193688 8037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:43:32.194906 8037 hybrid_clock.cc:648] HybridClock initialized: now 1790333012194836 us; error 45 us; skew 500 ppm
I20260925 10:43:32.195261 8037 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:43:32 dist-test-slave-dxdd krb5kdc[6958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333012, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:43:32.204447 8037 init.cc:377] Logged in from keytab as kudu/127.23.100.62@KRBTEST.COM (short username kudu)
I20260925 10:43:32.205912 8037 webserver.cc:460] Webserver started at http://127.23.100.62:46277/ using document root <none> and password file <none>
I20260925 10:43:32.206693 8037 fs_manager.cc:359] Metadata directory not provided
I20260925 10:43:32.206807 8037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:43:32.212225 8037 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260925 10:43:32.214869 8052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:43:32.216032 8037 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 10:43:32.216223 8037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/master-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/master-0/wal
uuid: "282881ffd90e4e42a826159c2fe1678d"
format_stamp: "Formatted at 2026-09-25 10:42:46 on dist-test-slave-dxdd"
server_key: "6d6e6f0211224999f567176237198484"
server_key_iv: "04f228acc47f06b9b946e5ed5ad5c5ee"
server_key_version: "encryptionkey@0"
I20260925 10:43:32.217160 8037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:43:32.231890 8037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:43:32.235613 8037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:43:32.235905 8037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:43:32.254243 8037 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.62:35471
I20260925 10:43:32.254261 8103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.62:35471 every 8 connection(s)
I20260925 10:43:32.255673 8037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
I20260925 10:43:32.262069 8104 sys_catalog.cc:263] Verifying existing consensus state
I20260925 10:43:32.262092 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 8037
I20260925 10:43:32.262951 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.1:43897
--local_ip_for_outbound_sockets=127.23.100.1
--tserver_master_addrs=127.23.100.62:35471
--webserver_port=40453
--webserver_interface=127.23.100.1
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab
--principal=kudu/127.23.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:38443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:43:32.265267 8104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d: Bootstrap starting.
I20260925 10:43:32.284759 8104 log.cc:824] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d: Log is configured to *not* fsync() on all Append() calls
I20260925 10:43:32.299938 8104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:43:32.300460 8104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d: Bootstrap complete.
I20260925 10:43:32.309223 8104 raft_consensus.cc:348] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "282881ffd90e4e42a826159c2fe1678d" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 35471 } }
I20260925 10:43:32.310098 8104 raft_consensus.cc:729] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 282881ffd90e4e42a826159c2fe1678d, State: Initialized, Role: FOLLOWER
I20260925 10:43:32.310750 8104 consensus_queue.cc:260] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "282881ffd90e4e42a826159c2fe1678d" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 35471 } }
I20260925 10:43:32.310999 8104 raft_consensus.cc:388] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:43:32.311125 8104 raft_consensus.cc:482] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:43:32.311311 8104 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d [term 2 FOLLOWER]: Advancing to term 3
I20260925 10:43:32.315042 8104 raft_consensus.cc:504] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "282881ffd90e4e42a826159c2fe1678d" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 35471 } }
I20260925 10:43:32.315527 8104 leader_election.cc:302] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 282881ffd90e4e42a826159c2fe1678d; no voters:
I20260925 10:43:32.316164 8104 leader_election.cc:288] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 10:43:32.316323 8108 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d [term 3 FOLLOWER]: Leader election won for term 3
I20260925 10:43:32.317376 8108 raft_consensus.cc:686] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d [term 3 LEADER]: Becoming Leader. State: Replica: 282881ffd90e4e42a826159c2fe1678d, State: Running, Role: LEADER
I20260925 10:43:32.317953 8108 consensus_queue.cc:237] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "282881ffd90e4e42a826159c2fe1678d" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 35471 } }
I20260925 10:43:32.318284 8104 sys_catalog.cc:564] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:43:32.320449 8109 sys_catalog.cc:455] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "282881ffd90e4e42a826159c2fe1678d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "282881ffd90e4e42a826159c2fe1678d" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 35471 } } }
I20260925 10:43:32.320879 8109 sys_catalog.cc:458] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d [sys.catalog]: This master's current role is: LEADER
I20260925 10:43:32.321346 8110 sys_catalog.cc:455] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 282881ffd90e4e42a826159c2fe1678d. Latest consensus state: current_term: 3 leader_uuid: "282881ffd90e4e42a826159c2fe1678d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "282881ffd90e4e42a826159c2fe1678d" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 35471 } } }
I20260925 10:43:32.321744 8110 sys_catalog.cc:458] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d [sys.catalog]: This master's current role is: LEADER
I20260925 10:43:32.323453 8114 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:43:32.332963 8114 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:43:32.334661 8114 catalog_manager.cc:1259] Loaded cluster ID: 78d8f9c6914141d38315fd6f65ef128f
I20260925 10:43:32.334798 8114 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:43:32.338658 8114 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:43:32.340459 8114 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d: Loaded TSK: 0
I20260925 10:43:32.340636 8114 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 282881ffd90e4e42a826159c2fe1678d: Loaded TSK: 1
F20260925 10:43:32.341737 8114 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fa41a15f9ad google::LogMessage::Fail() at ??:0
@ 0x7fa41a16288a google::LogMessage::SendToLog() at ??:0
@ 0x7fa41a15f4de google::LogMessage::Flush() at ??:0
@ 0x7fa41a160a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa41dec37d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
I20260925 10:43:32.352973 8126 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7fa41deb273e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fa42fb8e1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fa42fb8d783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fa42fc192a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fa42fc190c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fa42fc635e9 std::function<>::operator()() at ??:0
@ 0x7fa42fb91cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fa42fb8fc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fa42fc17610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fa42fc17439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fa41cdab345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa41cdae91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fa41cdae799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fa41cd89eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa4200636db start_thread at ??:0
@ 0x7fa418ff961f clone at ??:0
W20260925 10:43:32.658596 8106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:43:32.658872 8106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:43:32.658962 8106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:43:32.666016 8106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:43:32.666109 8106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:43:32.666231 8106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.1
I20260925 10:43:32.676061 8106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:38443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab
--principal=kudu/127.23.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.1:43897
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.1
--webserver_port=40453
--tserver_master_addrs=127.23.100.62:35471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.1
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:43:32.677199 8106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:43:32.678424 8106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:43:32.691358 8154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:43:32.691402 8152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:43:32.691448 8151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:43:32.692274 8106 server_base.cc:1034] running on GCE node
I20260925 10:43:32.692898 8106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:43:32.693948 8106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:43:32.695174 8106 hybrid_clock.cc:648] HybridClock initialized: now 1790333012695122 us; error 51 us; skew 500 ppm
I20260925 10:43:32.695501 8106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:43:32 dist-test-slave-dxdd krb5kdc[6958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333012, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:43:32.698642 8106 init.cc:377] Logged in from keytab as kudu/127.23.100.1@KRBTEST.COM (short username kudu)
I20260925 10:43:32.700053 8106 webserver.cc:460] Webserver started at http://127.23.100.1:40453/ using document root <none> and password file <none>
I20260925 10:43:32.700726 8106 fs_manager.cc:359] Metadata directory not provided
I20260925 10:43:32.700873 8106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:43:32.706053 8106 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:43:32.708778 8161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:43:32.710031 8106 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:43:32.710192 8106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-0/wal
uuid: "24c5c47272fe4d52b77ba1edc0d1e120"
format_stamp: "Formatted at 2026-09-25 10:42:47 on dist-test-slave-dxdd"
server_key: "aa9594b95baee257d2bd937db21816b5"
server_key_iv: "c678bb0bf00dafa9173c703922bdfd3c"
server_key_version: "encryptionkey@0"
I20260925 10:43:32.711167 8106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:43:32.724632 8106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:43:32.728695 8106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:43:32.728963 8106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:43:32.729884 8106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:43:32.731828 8168 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260925 10:43:32.737058 8106 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 10:43:32.737187 8106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260925 10:43:32.737322 8106 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 10:43:32.740257 8106 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 10:43:32.740356 8106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20260925 10:43:32.740664 8168 tablet_bootstrap.cc:492] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120: Bootstrap starting.
I20260925 10:43:32.768539 8168 log.cc:824] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120: Log is configured to *not* fsync() on all Append() calls
I20260925 10:43:32.778963 8106 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.1:43897
I20260925 10:43:32.778986 8275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.1:43897 every 8 connection(s)
I20260925 10:43:32.780175 8106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
I20260925 10:43:32.782256 8168 tablet_bootstrap.cc:492] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:43:32.784474 8168 tablet_bootstrap.cc:492] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120: Bootstrap complete.
I20260925 10:43:32.785006 8278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:32.783175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.1:51961)
0925 10:43:32.783864 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:32.784108 (+ 244us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":287,"threads_started":1}
W20260925 10:43:32.785696 8276 heartbeater.cc:644] Failed to heartbeat to 127.23.100.62:35471 (0 consecutive failures): Network error: Failed to ping master at 127.23.100.62:35471: Client connection negotiation failed: client connection to 127.23.100.62:35471: connect: Connection refused (error 111)
I20260925 10:43:32.785952 8168 ts_tablet_manager.cc:1397] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120: Time spent bootstrapping tablet: real 0.046s user 0.018s sys 0.023s
I20260925 10:43:32.786567 8278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:32.786214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.1:44213)
0925 10:43:32.786396 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:32.786466 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260925 10:43:32.786963 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 8106
I20260925 10:43:32.787693 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.2:39469
--local_ip_for_outbound_sockets=127.23.100.2
--tserver_master_addrs=127.23.100.62:35471
--webserver_port=42107
--webserver_interface=127.23.100.2
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab
--principal=kudu/127.23.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:38443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:43:32.791234 8278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:32.790890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.1:34527)
0925 10:43:32.791032 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:32.791116 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 10:43:32.791534 8276 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 10:43:32.796665 8168 raft_consensus.cc:348] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c5c47272fe4d52b77ba1edc0d1e120" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 43897 } } peers { permanent_uuid: "bd045bbe55db4881bcd6d93cbcc2a3af" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 39469 } } peers { permanent_uuid: "938822573aaf469abe7f961a07d8e36b" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 44349 } }
I20260925 10:43:32.797586 8168 raft_consensus.cc:729] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24c5c47272fe4d52b77ba1edc0d1e120, State: Initialized, Role: FOLLOWER
I20260925 10:43:32.798151 8168 consensus_queue.cc:260] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c5c47272fe4d52b77ba1edc0d1e120" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 43897 } } peers { permanent_uuid: "bd045bbe55db4881bcd6d93cbcc2a3af" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 39469 } } peers { permanent_uuid: "938822573aaf469abe7f961a07d8e36b" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 44349 } }
I20260925 10:43:32.799880 8168 ts_tablet_manager.cc:1428] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s
I20260925 10:43:32.799980 8278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:32.799538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.1:49773)
0925 10:43:32.799712 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:32.799799 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 10:43:32.800875 8278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:32.800587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.1:57341)
0925 10:43:32.800708 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:32.800769 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 10:43:33.115185 8279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:43:33.115437 8279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:43:33.115614 8279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:43:33.122761 8279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:43:33.122862 8279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:43:33.123054 8279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.2
I20260925 10:43:33.132696 8279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:38443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab
--principal=kudu/127.23.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.2:39469
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.2
--webserver_port=42107
--tserver_master_addrs=127.23.100.62:35471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.2
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:43:33.133924 8279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:43:33.135236 8279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:43:33.147399 8285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:43:33.148027 8286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:43:33.149163 8288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:43:33.149502 8279 server_base.cc:1034] running on GCE node
I20260925 10:43:33.150055 8279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:43:33.150949 8279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:43:33.152144 8279 hybrid_clock.cc:648] HybridClock initialized: now 1790333013152097 us; error 40 us; skew 500 ppm
I20260925 10:43:33.152447 8279 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:43:33 dist-test-slave-dxdd krb5kdc[6958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333013, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:43:33.155635 8279 init.cc:377] Logged in from keytab as kudu/127.23.100.2@KRBTEST.COM (short username kudu)
I20260925 10:43:33.156915 8279 webserver.cc:460] Webserver started at http://127.23.100.2:42107/ using document root <none> and password file <none>
I20260925 10:43:33.157536 8279 fs_manager.cc:359] Metadata directory not provided
I20260925 10:43:33.157630 8279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:43:33.162858 8279 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 10:43:33.165473 8295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:43:33.166807 8279 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:43:33.166980 8279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-1/wal
uuid: "bd045bbe55db4881bcd6d93cbcc2a3af"
format_stamp: "Formatted at 2026-09-25 10:42:47 on dist-test-slave-dxdd"
server_key: "b0535cac90591377fca569b392e5d358"
server_key_iv: "b08fc578239d0f300681f7df6f7ad797"
server_key_version: "encryptionkey@0"
I20260925 10:43:33.167950 8279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:43:33.186502 8279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:43:33.190683 8279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:43:33.190968 8279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:43:33.191928 8279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:43:33.193197 8279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:43:33.193323 8279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:43:33.193463 8279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:43:33.193554 8279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:43:33.226111 8279 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.2:39469
I20260925 10:43:33.226197 8407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.2:39469 every 8 connection(s)
I20260925 10:43:33.227591 8279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
I20260925 10:43:33.232511 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:33.230560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.2:35919)
0925 10:43:33.231398 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:33.231627 (+ 229us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":329,"threads_started":1}
W20260925 10:43:33.233079 8408 heartbeater.cc:644] Failed to heartbeat to 127.23.100.62:35471 (0 consecutive failures): Network error: Failed to ping master at 127.23.100.62:35471: Client connection negotiation failed: client connection to 127.23.100.62:35471: connect: Connection refused (error 111)
I20260925 10:43:33.233335 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 8279
I20260925 10:43:33.233908 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:33.233610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.2:44967)
0925 10:43:33.233735 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:33.233794 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 10:43:33.234128 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.3:44349
--local_ip_for_outbound_sockets=127.23.100.3
--tserver_master_addrs=127.23.100.62:35471
--webserver_port=40667
--webserver_interface=127.23.100.3
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab
--principal=kudu/127.23.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:38443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:43:33.234799 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:33.234519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.2:42067)
0925 10:43:33.234637 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:33.234702 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 10:43:33.235116 8408 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 10:43:33.548017 8411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:43:33.548266 8411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:43:33.548370 8411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:43:33.555226 8411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:43:33.555320 8411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:43:33.555517 8411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.3
I20260925 10:43:33.564893 8411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:38443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab
--principal=kudu/127.23.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.3:44349
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.3
--webserver_port=40667
--tserver_master_addrs=127.23.100.62:35471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.3
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:43:33.566041 8411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:43:33.567269 8411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:43:33.581022 8419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:43:33.581149 8416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:43:33.581171 8417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:43:33.581853 8411 server_base.cc:1034] running on GCE node
I20260925 10:43:33.582448 8411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:43:33.583280 8411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:43:33.584474 8411 hybrid_clock.cc:648] HybridClock initialized: now 1790333013584430 us; error 41 us; skew 500 ppm
I20260925 10:43:33.584729 8411 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:43:33 dist-test-slave-dxdd krb5kdc[6958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333013, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:43:33.587729 8411 init.cc:377] Logged in from keytab as kudu/127.23.100.3@KRBTEST.COM (short username kudu)
I20260925 10:43:33.588972 8411 webserver.cc:460] Webserver started at http://127.23.100.3:40667/ using document root <none> and password file <none>
I20260925 10:43:33.589576 8411 fs_manager.cc:359] Metadata directory not provided
I20260925 10:43:33.589671 8411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:43:33.594635 8411 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:43:33.597311 8426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:43:33.598654 8411 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:43:33.598795 8411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-2/wal
uuid: "938822573aaf469abe7f961a07d8e36b"
format_stamp: "Formatted at 2026-09-25 10:42:48 on dist-test-slave-dxdd"
server_key: "c8114d8716ce625d6069bd7f1d2f73e3"
server_key_iv: "99b959de31f0f6579b68096f16f12e02"
server_key_version: "encryptionkey@0"
I20260925 10:43:33.599718 8411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:43:33.626464 8411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:43:33.630679 8411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:43:33.630978 8411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:43:33.631909 8411 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:43:33.633191 8411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:43:33.633277 8411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:43:33.633347 8411 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:43:33.633399 8411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:43:33.665531 8411 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.3:44349
I20260925 10:43:33.665714 8538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.3:44349 every 8 connection(s)
I20260925 10:43:33.667053 8411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
I20260925 10:43:33.667465 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 8411
I20260925 10:43:33.671908 8542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:33.670010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.3:48327)
0925 10:43:33.670752 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:33.671026 (+ 274us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":343,"threads_started":1}
W20260925 10:43:33.672555 8539 heartbeater.cc:644] Failed to heartbeat to 127.23.100.62:35471 (0 consecutive failures): Network error: Failed to ping master at 127.23.100.62:35471: Client connection negotiation failed: client connection to 127.23.100.62:35471: connect: Connection refused (error 111)
I20260925 10:43:33.673470 8542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:33.673057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.3:60991)
0925 10:43:33.673305 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:33.673371 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
I20260925 10:43:33.674239 8542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:33.673943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.3:41031)
0925 10:43:33.674068 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:33.674137 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 10:43:33.674506 8539 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 10:43:33.802452 8543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:33.801689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.1:36399)
0925 10:43:33.802112 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:33.802187 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":204,"threads_started":1}
I20260925 10:43:34.236500 8544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:34.235654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.2:44987)
0925 10:43:34.236173 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:34.236286 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":288,"threads_started":1}
I20260925 10:43:34.606600 8545 raft_consensus.cc:482] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:43:34.606873 8545 raft_consensus.cc:504] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c5c47272fe4d52b77ba1edc0d1e120" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 43897 } } peers { permanent_uuid: "bd045bbe55db4881bcd6d93cbcc2a3af" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 39469 } } peers { permanent_uuid: "938822573aaf469abe7f961a07d8e36b" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 44349 } }
I20260925 10:43:34.608489 8545 leader_election.cc:288] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bd045bbe55db4881bcd6d93cbcc2a3af (127.23.100.2:39469), 938822573aaf469abe7f961a07d8e36b (127.23.100.3:44349)
Sep 25 10:43:34 dist-test-slave-dxdd krb5kdc[6958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333012, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.1@KRBTEST.COM for kudu/127.23.100.2@KRBTEST.COM
Sep 25 10:43:34 dist-test-slave-dxdd krb5kdc[6958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333012, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.1@KRBTEST.COM for kudu/127.23.100.3@KRBTEST.COM
I20260925 10:43:34.643072 8547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:34.608812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.3:44349 (local address 127.23.100.1:34251)
0925 10:43:34.621722 (+ 12910us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:34.621787 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 10:43:34.622815 (+ 1028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:43:34.626139 (+ 3324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:43:34.626145 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:43:34.627171 (+ 1026us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:43:34.627872 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:34.627892 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:34.629650 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:34.629655 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:34.631458 (+ 1803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:34.631470 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:34.632479 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:34.632485 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:34.632730 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:34.633494 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:43:34.633518 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:43:34.636281 (+ 2763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:43:34.639520 (+ 3239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:43:34.639529 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:43:34.639545 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:43:34.640159 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:43:34.640697 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:43:34.640702 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:43:34.640706 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:43:34.641007 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:43:34.641683 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:43:34.641688 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:43:34.641933 (+ 245us) client_negotiation.cc:769] Sending connection context
0925 10:43:34.642137 (+ 204us) client_negotiation.cc:240] Negotiation successful
0925 10:43:34.642596 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12726,"mutex_wait_us":179,"spinlock_wait_cycles":25856,"thread_start_us":14525,"threads_started":1}
I20260925 10:43:34.643409 8549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:34.608980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:34251 (local address 127.23.100.3:44349)
0925 10:43:34.623896 (+ 14916us) server_negotiation.cc:206] Beginning negotiation
0925 10:43:34.623912 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 10:43:34.623954 (+ 42us) server_negotiation.cc:407] Connection header received
0925 10:43:34.625299 (+ 1345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:43:34.625308 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:43:34.625692 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:43:34.626018 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:43:34.628057 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:34.629379 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:34.631583 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:34.632253 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:34.632455 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:34.636525 (+ 4070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:43:34.636531 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:43:34.636543 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:43:34.636576 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:43:34.639379 (+ 2803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:43:34.640351 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:43:34.640360 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:43:34.640369 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:43:34.640567 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:43:34.641117 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:43:34.641121 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:43:34.641124 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:43:34.641571 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:43:34.641725 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 10:43:34.642475 (+ 750us) server_negotiation.cc:299] Negotiation successful
0925 10:43:34.642850 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14748,"thread_start_us":14656,"threads_started":1}
I20260925 10:43:34.643826 8548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:34.608958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:57241 (local address 127.23.100.2:39469)
0925 10:43:34.623571 (+ 14613us) server_negotiation.cc:206] Beginning negotiation
0925 10:43:34.623593 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 10:43:34.623842 (+ 249us) server_negotiation.cc:407] Connection header received
0925 10:43:34.625413 (+ 1571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:43:34.625421 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:43:34.625733 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:43:34.626049 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:43:34.628066 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:34.629341 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:34.631608 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:34.632237 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:34.632413 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:34.636220 (+ 3807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:43:34.636230 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:43:34.636248 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:43:34.636296 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:43:34.639389 (+ 3093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:43:34.640368 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:43:34.640373 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:43:34.640378 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:43:34.640550 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:43:34.641781 (+ 1231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:43:34.641784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:43:34.641786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:43:34.642318 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:43:34.642511 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0925 10:43:34.643035 (+ 524us) server_negotiation.cc:299] Negotiation successful
0925 10:43:34.643453 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14358,"thread_start_us":14652,"threads_started":1}
I20260925 10:43:34.644170 8546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:34.608838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.2:39469 (local address 127.23.100.1:57241)
0925 10:43:34.623642 (+ 14804us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:34.623715 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0925 10:43:34.623826 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:43:34.626122 (+ 2296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:43:34.626133 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:43:34.627171 (+ 1038us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:43:34.627868 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:34.627886 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:34.629655 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:34.629659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:34.631456 (+ 1797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:34.631476 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:34.632434 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:34.632441 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:34.632728 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:34.633568 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:43:34.633585 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:43:34.635896 (+ 2311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:43:34.639561 (+ 3665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:43:34.639564 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:43:34.639567 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:43:34.640199 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:43:34.640697 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:43:34.640702 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:43:34.640706 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:43:34.641007 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:43:34.642480 (+ 1473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:43:34.642485 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:43:34.642632 (+ 147us) client_negotiation.cc:769] Sending connection context
0925 10:43:34.642760 (+ 128us) client_negotiation.cc:240] Negotiation successful
0925 10:43:34.643022 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14677,"mutex_wait_us":193,"spinlock_wait_cycles":16256,"thread_start_us":13487,"threads_started":1}
W20260925 10:43:34.646512 8164 leader_election.cc:341] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 938822573aaf469abe7f961a07d8e36b (127.23.100.3:44349): Not found: Tablet not found: f955ef6fb4024db885e3795d39f1c85f
W20260925 10:43:34.646773 8163 leader_election.cc:341] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bd045bbe55db4881bcd6d93cbcc2a3af (127.23.100.2:39469): Not found: Tablet not found: f955ef6fb4024db885e3795d39f1c85f
I20260925 10:43:34.646907 8163 leader_election.cc:302] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24c5c47272fe4d52b77ba1edc0d1e120; no voters: 938822573aaf469abe7f961a07d8e36b, bd045bbe55db4881bcd6d93cbcc2a3af
I20260925 10:43:34.647344 8545 raft_consensus.cc:2726] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 10:43:34.675699 8550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:34.674934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.3:42317)
0925 10:43:34.675355 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:34.675484 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":211,"threads_started":1}
I20260925 10:43:34.803845 8546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:34.803430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.1:59985)
0925 10:43:34.803599 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:34.803700 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 10:43:35.238111 8551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:35.237428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.2:54735)
0925 10:43:35.237884 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:35.237961 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":264,"threads_started":1}
I20260925 10:43:35.677554 8552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:35.676691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.3:57453)
0925 10:43:35.677232 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:35.677339 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":276,"threads_started":1}
I20260925 10:43:35.805449 8553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:35.804669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.1:37617)
0925 10:43:35.805192 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:35.805273 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":250,"threads_started":1}
I20260925 10:43:36.149667 8554 raft_consensus.cc:482] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:43:36.149849 8554 raft_consensus.cc:504] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c5c47272fe4d52b77ba1edc0d1e120" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 43897 } } peers { permanent_uuid: "bd045bbe55db4881bcd6d93cbcc2a3af" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 39469 } } peers { permanent_uuid: "938822573aaf469abe7f961a07d8e36b" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 44349 } }
I20260925 10:43:36.150493 8554 leader_election.cc:288] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bd045bbe55db4881bcd6d93cbcc2a3af (127.23.100.2:39469), 938822573aaf469abe7f961a07d8e36b (127.23.100.3:44349)
W20260925 10:43:36.151352 8163 leader_election.cc:341] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bd045bbe55db4881bcd6d93cbcc2a3af (127.23.100.2:39469): Not found: Tablet not found: f955ef6fb4024db885e3795d39f1c85f
W20260925 10:43:36.151543 8164 leader_election.cc:341] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 938822573aaf469abe7f961a07d8e36b (127.23.100.3:44349): Not found: Tablet not found: f955ef6fb4024db885e3795d39f1c85f
I20260925 10:43:36.151747 8164 leader_election.cc:302] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24c5c47272fe4d52b77ba1edc0d1e120; no voters: 938822573aaf469abe7f961a07d8e36b, bd045bbe55db4881bcd6d93cbcc2a3af
I20260925 10:43:36.152001 8554 raft_consensus.cc:2726] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 10:43:36.239578 8555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:36.238946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.2:36625)
0925 10:43:36.239349 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:36.239423 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":218,"threads_started":1}
I20260925 10:43:36.679328 8556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:36.678670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.3:59209)
0925 10:43:36.679077 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:36.679173 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":203,"threads_started":1}
I20260925 10:43:36.807056 8557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:36.806430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.1:34075)
0925 10:43:36.806824 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:36.806902 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":201,"threads_started":1}
I20260925 10:43:37.241034 8558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:37.240387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.2:38779)
0925 10:43:37.240787 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:37.240866 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":199,"threads_started":1}
I20260925 10:43:37.680841 8559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:37.680106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.3:60837)
0925 10:43:37.680596 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:37.680670 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":243,"threads_started":1}
I20260925 10:43:37.808687 8560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:37.808037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.1:57251)
0925 10:43:37.808443 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:37.808519 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":212,"threads_started":1}
I20260925 10:43:38.242555 8561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:38.241866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.2:41613)
0925 10:43:38.242261 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:38.242393 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":209,"threads_started":1}
I20260925 10:43:38.682237 8562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:38.681608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.3:35811)
0925 10:43:38.681998 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:38.682072 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":206,"threads_started":1}
I20260925 10:43:38.810290 8563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:38.809586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.1:44285)
0925 10:43:38.810018 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:38.810115 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":244,"threads_started":1}
I20260925 10:43:39.243979 8564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:39.243314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.2:39537)
0925 10:43:39.243734 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:39.243814 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":217,"threads_started":1}
I20260925 10:43:39.370622 8565 raft_consensus.cc:482] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:43:39.370851 8565 raft_consensus.cc:504] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c5c47272fe4d52b77ba1edc0d1e120" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 43897 } } peers { permanent_uuid: "bd045bbe55db4881bcd6d93cbcc2a3af" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 39469 } } peers { permanent_uuid: "938822573aaf469abe7f961a07d8e36b" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 44349 } }
I20260925 10:43:39.371503 8565 leader_election.cc:288] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bd045bbe55db4881bcd6d93cbcc2a3af (127.23.100.2:39469), 938822573aaf469abe7f961a07d8e36b (127.23.100.3:44349)
W20260925 10:43:39.372386 8163 leader_election.cc:341] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bd045bbe55db4881bcd6d93cbcc2a3af (127.23.100.2:39469): Not found: Tablet not found: f955ef6fb4024db885e3795d39f1c85f
W20260925 10:43:39.372517 8164 leader_election.cc:341] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 938822573aaf469abe7f961a07d8e36b (127.23.100.3:44349): Not found: Tablet not found: f955ef6fb4024db885e3795d39f1c85f
I20260925 10:43:39.372679 8164 leader_election.cc:302] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24c5c47272fe4d52b77ba1edc0d1e120; no voters: 938822573aaf469abe7f961a07d8e36b, bd045bbe55db4881bcd6d93cbcc2a3af
I20260925 10:43:39.372917 8565 raft_consensus.cc:2726] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 10:43:39.683696 8566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:39.683037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.3:46601)
0925 10:43:39.683459 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:39.683532 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":235,"threads_started":1}
I20260925 10:43:39.811926 8567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:39.811148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.1:45893)
0925 10:43:39.811624 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:39.811736 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":219,"threads_started":1}
I20260925 10:43:40.245621 8568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:40.244863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.2:60063)
0925 10:43:40.245370 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:40.245456 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":232,"threads_started":1}
I20260925 10:43:40.685628 8569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:40.684727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.3:44469)
0925 10:43:40.685364 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:40.685463 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":296,"threads_started":1}
I20260925 10:43:40.813506 8570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:40.812833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.1:38183)
0925 10:43:40.813261 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:40.813354 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":225,"threads_started":1}
I20260925 10:43:41.247267 8571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:41.246584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.2:48887)
0925 10:43:41.246986 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:41.247080 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":207,"threads_started":1}
I20260925 10:43:41.687237 8572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:41.686605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.3:40639)
0925 10:43:41.686984 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:41.687064 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":195,"threads_started":1}
I20260925 10:43:41.815076 8573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:41.814420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.1:53969)
0925 10:43:41.814824 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:41.814903 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":209,"threads_started":1}
I20260925 10:43:42.248803 8574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:42.248091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.2:35189)
0925 10:43:42.248564 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:42.248648 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":250,"threads_started":1}
I20260925 10:43:42.688668 8575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:42.688008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.3:36479)
0925 10:43:42.688448 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:42.688523 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":241,"threads_started":1}
I20260925 10:43:42.816772 8576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:42.816062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.1:45485)
0925 10:43:42.816529 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:42.816608 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":264,"threads_started":1}
I20260925 10:43:43.250541 8577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:43.249760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.2:46529)
0925 10:43:43.250230 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:43.250371 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":208,"threads_started":1}
I20260925 10:43:43.690292 8578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:43.689619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.3:39779)
0925 10:43:43.690040 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:43.690113 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":250,"threads_started":1}
I20260925 10:43:43.818500 8579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:43.817675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.1:35199)
0925 10:43:43.818127 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:43.818236 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":239,"threads_started":1}
I20260925 10:43:43.938915 8580 raft_consensus.cc:482] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:43:43.939172 8580 raft_consensus.cc:504] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c5c47272fe4d52b77ba1edc0d1e120" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 43897 } } peers { permanent_uuid: "bd045bbe55db4881bcd6d93cbcc2a3af" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 39469 } } peers { permanent_uuid: "938822573aaf469abe7f961a07d8e36b" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 44349 } }
I20260925 10:43:43.939908 8580 leader_election.cc:288] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bd045bbe55db4881bcd6d93cbcc2a3af (127.23.100.2:39469), 938822573aaf469abe7f961a07d8e36b (127.23.100.3:44349)
W20260925 10:43:43.940821 8163 leader_election.cc:341] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bd045bbe55db4881bcd6d93cbcc2a3af (127.23.100.2:39469): Not found: Tablet not found: f955ef6fb4024db885e3795d39f1c85f
W20260925 10:43:43.941020 8164 leader_election.cc:341] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 938822573aaf469abe7f961a07d8e36b (127.23.100.3:44349): Not found: Tablet not found: f955ef6fb4024db885e3795d39f1c85f
I20260925 10:43:43.941179 8164 leader_election.cc:302] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24c5c47272fe4d52b77ba1edc0d1e120; no voters: 938822573aaf469abe7f961a07d8e36b, bd045bbe55db4881bcd6d93cbcc2a3af
I20260925 10:43:43.941431 8580 raft_consensus.cc:2726] T f955ef6fb4024db885e3795d39f1c85f P 24c5c47272fe4d52b77ba1edc0d1e120 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 10:43:44.252288 8581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:44.251475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.2:35769)
0925 10:43:44.252042 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:44.252128 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":316,"threads_started":1}
I20260925 10:43:44.692082 8582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:44.691302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.3:33019)
0925 10:43:44.691786 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:44.691875 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":240,"threads_started":1}
I20260925 10:43:44.819950 8583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:44.819286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.1:58073)
0925 10:43:44.819708 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:44.819796 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":218,"threads_started":1}
I20260925 10:43:45.253744 8584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:45.253105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.2:59183)
0925 10:43:45.253502 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:45.253583 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":207,"threads_started":1}
I20260925 10:43:45.693696 8585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:45.693006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.3:41111)
0925 10:43:45.693454 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:45.693529 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":244,"threads_started":1}
I20260925 10:43:45.821636 8586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:45.820821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.1:38849)
0925 10:43:45.821326 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:45.821415 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":251,"threads_started":1}
I20260925 10:43:46.255246 8587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:46.254576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.2:37947)
0925 10:43:46.254995 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:46.255087 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":213,"threads_started":1}
I20260925 10:43:46.695221 8588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:46.694576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.3:58451)
0925 10:43:46.694979 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:46.695067 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":206,"threads_started":1}
I20260925 10:43:46.823099 8589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:46.822446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.1:42333)
0925 10:43:46.822861 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:46.822938 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":225,"threads_started":1}
I20260925 10:43:47.256793 8590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:47.256155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.2:52795)
0925 10:43:47.256543 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:47.256615 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":205,"threads_started":1}
I20260925 10:43:47.696791 8591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:47.696101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.3:57253)
0925 10:43:47.696516 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:47.696601 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":226,"threads_started":1}
I20260925 10:43:47.824846 8592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:47.824131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.1:59537)
0925 10:43:47.824589 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:47.824677 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":241,"threads_started":1}
I20260925 10:43:48.258600 8593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:48.257841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.2:60235)
0925 10:43:48.258295 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:48.258432 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":223,"threads_started":1}
I20260925 10:43:48.671821 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 8106
I20260925 10:43:48.686858 8594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:48.673161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46686 (local address 127.23.100.1:43897)
0925 10:43:48.673792 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0925 10:43:48.673799 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:43:48.673826 (+ 27us) server_negotiation.cc:407] Connection header received
0925 10:43:48.673886 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:43:48.673892 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:43:48.674184 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:43:48.674391 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:43:48.675830 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:48.676729 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:48.678012 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:48.678647 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:48.678790 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:48.681505 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:43:48.681516 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:43:48.681529 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:43:48.681559 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:43:48.684075 (+ 2516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:43:48.684773 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:43:48.684779 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:43:48.684781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:43:48.684900 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:43:48.685294 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:43:48.685300 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:43:48.685303 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:43:48.685627 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:43:48.685950 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0925 10:43:48.686207 (+ 257us) server_negotiation.cc:299] Negotiation successful
0925 10:43:48.686499 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":303,"threads_started":1}
I20260925 10:43:48.688051 8271 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:43:48.698654 8596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:48.697840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35471 (local address 127.23.100.3:40151)
0925 10:43:48.698381 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:48.698492 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":305,"threads_started":1}
I20260925 10:43:48.777169 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 8106
I20260925 10:43:48.794734 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 8279
I20260925 10:43:48.807379 8597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:48.795496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41740 (local address 127.23.100.2:39469)
0925 10:43:48.796085 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0925 10:43:48.796089 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:43:48.796105 (+ 16us) server_negotiation.cc:407] Connection header received
0925 10:43:48.796172 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:43:48.796176 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:43:48.796231 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:43:48.796321 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:43:48.797541 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:48.798358 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:48.799585 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:48.799993 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:48.800084 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:48.802575 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:43:48.802583 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:43:48.802591 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:43:48.802632 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:43:48.805124 (+ 2492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:43:48.805779 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:43:48.805782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:43:48.805784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:43:48.805905 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:43:48.806203 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:43:48.806206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:43:48.806208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:43:48.806504 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:43:48.806704 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 10:43:48.806920 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 10:43:48.807084 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":383,"threads_started":1}
I20260925 10:43:48.808056 8403 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:43:48.889447 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 8279
I20260925 10:43:48.906263 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 8411
I20260925 10:43:48.919911 8599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:48.907098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33952 (local address 127.23.100.3:44349)
0925 10:43:48.907701 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0925 10:43:48.907705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:43:48.907719 (+ 14us) server_negotiation.cc:407] Connection header received
0925 10:43:48.907770 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:43:48.907774 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:43:48.907835 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:43:48.907954 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:43:48.909326 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:48.910093 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:48.911491 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:48.911925 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:48.912060 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:48.914589 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:43:48.914594 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:43:48.914597 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:43:48.914647 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:43:48.917175 (+ 2528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:43:48.917824 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:43:48.917831 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:43:48.917835 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:43:48.917989 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:43:48.918499 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:43:48.918506 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:43:48.918509 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:43:48.918757 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:43:48.919037 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0925 10:43:48.919324 (+ 287us) server_negotiation.cc:299] Negotiation successful
0925 10:43:48.919535 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":356,"threads_started":1}
I20260925 10:43:48.920614 8534 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:43:49.003545 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 8411
2026-09-25T10:43:49Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64263 ms)
[----------] 30 tests from SecurityITest (288906 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:43:49 dist-test-slave-dxdd krb5kdc[8603](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:43:49 dist-test-slave-dxdd krb5kdc[8603](info): set up 2 sockets
Sep 25 10:43:49 dist-test-slave-dxdd krb5kdc[8603](info): commencing operation
krb5kdc: starting...
W20260925 10:43:50.086577 23952 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Sep 25 10:43:50 dist-test-slave-dxdd krb5kdc[8603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:43:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:43:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
I20260925 10:43:50.318849 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:45657
--webserver_interface=127.23.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:37745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.62:45657
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:43:50.635725 8616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:43:50.636045 8616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:43:50.636212 8616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:43:50.643371 8616 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:43:50.643522 8616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:43:50.643640 8616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:43:50.643738 8616 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:43:50.643821 8616 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:43:50.653354 8616 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:37745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.62:45657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:45657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:43:50.654678 8616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:43:50.655970 8616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:43:50.667730 8621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:43:50.667690 8624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:43:50.667781 8622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:43:50.668594 8616 server_base.cc:1034] running on GCE node
I20260925 10:43:50.669330 8616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:43:50.670791 8616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:43:50.672017 8616 hybrid_clock.cc:648] HybridClock initialized: now 1790333030671982 us; error 43 us; skew 500 ppm
I20260925 10:43:50.672360 8616 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:43:50 dist-test-slave-dxdd krb5kdc[8603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333030, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:43:50.675338 8616 init.cc:377] Logged in from keytab as kudu/127.23.100.62@KRBTEST.COM (short username kudu)
I20260925 10:43:50.676759 8616 webserver.cc:460] Webserver started at http://127.23.100.62:39205/ using document root <none> and password file <none>
I20260925 10:43:50.677404 8616 fs_manager.cc:359] Metadata directory not provided
I20260925 10:43:50.677544 8616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:43:50.677838 8616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:43:50.680836 8616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/master-0/data/instance:
uuid: "acfdd5b38c3f4aa8af5afd47c8de3226"
format_stamp: "Formatted at 2026-09-25 10:43:50 on dist-test-slave-dxdd"
server_key: "a0f0daf55ec627cb0d6ebb749f89a85d"
server_key_iv: "a494571f3e165b1eddbbc851e0936e51"
server_key_version: "encryptionkey@0"
I20260925 10:43:50.681551 8616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/master-0/wal/instance:
uuid: "acfdd5b38c3f4aa8af5afd47c8de3226"
format_stamp: "Formatted at 2026-09-25 10:43:50 on dist-test-slave-dxdd"
server_key: "a0f0daf55ec627cb0d6ebb749f89a85d"
server_key_iv: "a494571f3e165b1eddbbc851e0936e51"
server_key_version: "encryptionkey@0"
I20260925 10:43:50.686355 8616 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260925 10:43:50.689455 8631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:43:50.690804 8616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260925 10:43:50.690980 8616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/master-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/master-0/wal
uuid: "acfdd5b38c3f4aa8af5afd47c8de3226"
format_stamp: "Formatted at 2026-09-25 10:43:50 on dist-test-slave-dxdd"
server_key: "a0f0daf55ec627cb0d6ebb749f89a85d"
server_key_iv: "a494571f3e165b1eddbbc851e0936e51"
server_key_version: "encryptionkey@0"
I20260925 10:43:50.691172 8616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:43:50.713469 8616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:43:50.717057 8616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:43:50.717322 8616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:43:50.735229 8616 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.62:45657
I20260925 10:43:50.735257 8682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.62:45657 every 8 connection(s)
I20260925 10:43:50.736439 8616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
I20260925 10:43:50.739820 8683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:43:50.740950 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 8616
I20260925 10:43:50.741173 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/master-0/wal/instance
I20260925 10:43:50.741806 23952 external_mini_cluster.cc:1428] Setting key 8adaf0df74ec0de12744915eb5a38277
I20260925 10:43:50.752990 8683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acfdd5b38c3f4aa8af5afd47c8de3226: Bootstrap starting.
I20260925 10:43:50.755980 8683 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P acfdd5b38c3f4aa8af5afd47c8de3226: Neither blocks nor log segments found. Creating new log.
I20260925 10:43:50.756944 8683 log.cc:824] T 00000000000000000000000000000000 P acfdd5b38c3f4aa8af5afd47c8de3226: Log is configured to *not* fsync() on all Append() calls
I20260925 10:43:50.760123 8683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acfdd5b38c3f4aa8af5afd47c8de3226: No bootstrap required, opened a new log
Sep 25 10:43:50 dist-test-slave-dxdd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:43:50.767759 8683 raft_consensus.cc:348] T 00000000000000000000000000000000 P acfdd5b38c3f4aa8af5afd47c8de3226 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acfdd5b38c3f4aa8af5afd47c8de3226" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 45657 } }
I20260925 10:43:50.768126 8683 raft_consensus.cc:374] T 00000000000000000000000000000000 P acfdd5b38c3f4aa8af5afd47c8de3226 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:43:50.768273 8683 raft_consensus.cc:729] T 00000000000000000000000000000000 P acfdd5b38c3f4aa8af5afd47c8de3226 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acfdd5b38c3f4aa8af5afd47c8de3226, State: Initialized, Role: FOLLOWER
I20260925 10:43:50.768915 8683 consensus_queue.cc:260] T 00000000000000000000000000000000 P acfdd5b38c3f4aa8af5afd47c8de3226 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acfdd5b38c3f4aa8af5afd47c8de3226" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 45657 } }
I20260925 10:43:50.769172 8683 raft_consensus.cc:388] T 00000000000000000000000000000000 P acfdd5b38c3f4aa8af5afd47c8de3226 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:43:50.769320 8683 raft_consensus.cc:482] T 00000000000000000000000000000000 P acfdd5b38c3f4aa8af5afd47c8de3226 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:43:50.769512 8683 raft_consensus.cc:3037] T 00000000000000000000000000000000 P acfdd5b38c3f4aa8af5afd47c8de3226 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:43:50.769847 8686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:50.743843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59914 (local address 127.23.100.62:45657)
0925 10:43:50.744675 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0925 10:43:50.744687 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:43:50.744723 (+ 36us) server_negotiation.cc:407] Connection header received
0925 10:43:50.746264 (+ 1541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:43:50.746287 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:43:50.746698 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:43:50.746982 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:43:50.748695 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:50.749963 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:50.751813 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:50.752442 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:50.752565 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:50.762456 (+ 9891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:43:50.762465 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:43:50.762479 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:43:50.762517 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:43:50.765690 (+ 3173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:43:50.766426 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:43:50.766442 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:43:50.766447 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:43:50.766640 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:43:50.767042 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:43:50.767046 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:43:50.767048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:43:50.767520 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:43:50.767860 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0925 10:43:50.768235 (+ 375us) server_negotiation.cc:299] Negotiation successful
0925 10:43:50.768722 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":398,"threads_started":1}
I20260925 10:43:50.771850 8683 raft_consensus.cc:504] T 00000000000000000000000000000000 P acfdd5b38c3f4aa8af5afd47c8de3226 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acfdd5b38c3f4aa8af5afd47c8de3226" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 45657 } }
I20260925 10:43:50.772339 8683 leader_election.cc:302] T 00000000000000000000000000000000 P acfdd5b38c3f4aa8af5afd47c8de3226 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: acfdd5b38c3f4aa8af5afd47c8de3226; no voters:
I20260925 10:43:50.772969 8683 leader_election.cc:288] T 00000000000000000000000000000000 P acfdd5b38c3f4aa8af5afd47c8de3226 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:43:50.773111 8688 raft_consensus.cc:2781] T 00000000000000000000000000000000 P acfdd5b38c3f4aa8af5afd47c8de3226 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:43:50.773814 8688 raft_consensus.cc:686] T 00000000000000000000000000000000 P acfdd5b38c3f4aa8af5afd47c8de3226 [term 1 LEADER]: Becoming Leader. State: Replica: acfdd5b38c3f4aa8af5afd47c8de3226, State: Running, Role: LEADER
I20260925 10:43:50.774422 8688 consensus_queue.cc:237] T 00000000000000000000000000000000 P acfdd5b38c3f4aa8af5afd47c8de3226 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acfdd5b38c3f4aa8af5afd47c8de3226" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 45657 } }
I20260925 10:43:50.775040 8683 sys_catalog.cc:564] T 00000000000000000000000000000000 P acfdd5b38c3f4aa8af5afd47c8de3226 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:43:50.777171 8690 sys_catalog.cc:455] T 00000000000000000000000000000000 P acfdd5b38c3f4aa8af5afd47c8de3226 [sys.catalog]: SysCatalogTable state changed. Reason: New leader acfdd5b38c3f4aa8af5afd47c8de3226. Latest consensus state: current_term: 1 leader_uuid: "acfdd5b38c3f4aa8af5afd47c8de3226" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acfdd5b38c3f4aa8af5afd47c8de3226" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 45657 } } }
I20260925 10:43:50.777582 8690 sys_catalog.cc:458] T 00000000000000000000000000000000 P acfdd5b38c3f4aa8af5afd47c8de3226 [sys.catalog]: This master's current role is: LEADER
I20260925 10:43:50.777148 8689 sys_catalog.cc:455] T 00000000000000000000000000000000 P acfdd5b38c3f4aa8af5afd47c8de3226 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "acfdd5b38c3f4aa8af5afd47c8de3226" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acfdd5b38c3f4aa8af5afd47c8de3226" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 45657 } } }
I20260925 10:43:50.777993 8689 sys_catalog.cc:458] T 00000000000000000000000000000000 P acfdd5b38c3f4aa8af5afd47c8de3226 [sys.catalog]: This master's current role is: LEADER
I20260925 10:43:50.780791 8695 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:43:50.785562 8695 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:43:50.794341 8695 catalog_manager.cc:1347] Generated new cluster ID: 2849b69e6e8f406fa25f568f64d716a7
I20260925 10:43:50.794446 8695 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:43:50.821797 8695 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:43:50.823571 8695 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:43:50.836256 8695 catalog_manager.cc:5868] T 00000000000000000000000000000000 P acfdd5b38c3f4aa8af5afd47c8de3226: Generated new TSK 0
I20260925 10:43:50.837203 8695 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab.
Entry for principal kudu/127.23.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab.
I20260925 10:43:50.905944 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.1:0
--local_ip_for_outbound_sockets=127.23.100.1
--webserver_interface=127.23.100.1
--webserver_port=0
--tserver_master_addrs=127.23.100.62:45657
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab
--principal=kudu/127.23.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:37745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:43:51.230459 8709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:43:51.230708 8709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:43:51.230872 8709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:43:51.237699 8709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:43:51.237797 8709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:43:51.237932 8709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.1
I20260925 10:43:51.247145 8709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:37745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab
--principal=kudu/127.23.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.1
--webserver_port=0
--tserver_master_addrs=127.23.100.62:45657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.1
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:43:51.248302 8709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:43:51.249547 8709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:43:51.262368 8714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:43:51.262380 8715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:43:51.262404 8717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:43:51.263430 8709 server_base.cc:1034] running on GCE node
I20260925 10:43:51.264021 8709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:43:51.264811 8709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:43:51.266026 8709 hybrid_clock.cc:648] HybridClock initialized: now 1790333031265964 us; error 29 us; skew 500 ppm
I20260925 10:43:51.266404 8709 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:43:51 dist-test-slave-dxdd krb5kdc[8603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333031, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:43:51.269366 8709 init.cc:377] Logged in from keytab as kudu/127.23.100.1@KRBTEST.COM (short username kudu)
I20260925 10:43:51.270745 8709 webserver.cc:460] Webserver started at http://127.23.100.1:34595/ using document root <none> and password file <none>
I20260925 10:43:51.271397 8709 fs_manager.cc:359] Metadata directory not provided
I20260925 10:43:51.271539 8709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:43:51.271839 8709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:43:51.274521 8709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/ts-0/data/instance:
uuid: "5eeeb702631649a8ae5270adcd356e87"
format_stamp: "Formatted at 2026-09-25 10:43:51 on dist-test-slave-dxdd"
server_key: "17b87a685d7ec240efbe63ea7fd90cf9"
server_key_iv: "047478a4f7160b4e2a5bfe3be7292fbd"
server_key_version: "encryptionkey@0"
I20260925 10:43:51.275317 8709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/ts-0/wal/instance:
uuid: "5eeeb702631649a8ae5270adcd356e87"
format_stamp: "Formatted at 2026-09-25 10:43:51 on dist-test-slave-dxdd"
server_key: "17b87a685d7ec240efbe63ea7fd90cf9"
server_key_iv: "047478a4f7160b4e2a5bfe3be7292fbd"
server_key_version: "encryptionkey@0"
I20260925 10:43:51.280803 8709 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 10:43:51.284304 8724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:43:51.285470 8709 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:43:51.285640 8709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/ts-0/wal
uuid: "5eeeb702631649a8ae5270adcd356e87"
format_stamp: "Formatted at 2026-09-25 10:43:51 on dist-test-slave-dxdd"
server_key: "17b87a685d7ec240efbe63ea7fd90cf9"
server_key_iv: "047478a4f7160b4e2a5bfe3be7292fbd"
server_key_version: "encryptionkey@0"
I20260925 10:43:51.285856 8709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:43:51.300520 8709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:43:51.304490 8709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:43:51.304744 8709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:43:51.305583 8709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:43:51.306862 8709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:43:51.306939 8709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:43:51.307024 8709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:43:51.307065 8709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:43:51.339417 8709 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.1:46059
I20260925 10:43:51.339461 8836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.1:46059 every 8 connection(s)
I20260925 10:43:51.340703 8709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
I20260925 10:43:51.341246 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 8709
I20260925 10:43:51.341470 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/ts-0/wal/instance
I20260925 10:43:51.342291 23952 external_mini_cluster.cc:1428] Setting key 3d9250427754e86ac59449c055f326d3
Sep 25 10:43:51 dist-test-slave-dxdd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333031, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.1@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:43:51.362260 8839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:51.343437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:60763 (local address 127.23.100.62:45657)
0925 10:43:51.343889 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0925 10:43:51.343894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:43:51.345127 (+ 1233us) server_negotiation.cc:407] Connection header received
0925 10:43:51.347042 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:43:51.347050 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:43:51.347133 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:43:51.347271 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:43:51.348711 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:51.349901 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:51.351466 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:51.352158 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:51.352256 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:51.356424 (+ 4168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:43:51.356430 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:43:51.356433 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:43:51.356463 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:43:51.359193 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:43:51.359927 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:43:51.359937 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:43:51.359939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:43:51.360074 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:43:51.360504 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:43:51.360516 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:43:51.360519 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:43:51.360767 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:43:51.360937 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 10:43:51.361844 (+ 907us) server_negotiation.cc:299] Negotiation successful
0925 10:43:51.362011 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":240,"threads_started":1}
I20260925 10:43:51.363725 8840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:51.344155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:45657 (local address 127.23.100.1:60763)
0925 10:43:51.344957 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:51.345010 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 10:43:51.346746 (+ 1736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:43:51.347481 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:43:51.347489 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:43:51.347971 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:43:51.348544 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:51.348560 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:51.350054 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:51.350058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:51.351330 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:51.351341 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:51.352378 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:51.352387 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:51.352707 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:51.353455 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:43:51.353486 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:43:51.356198 (+ 2712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:43:51.359342 (+ 3144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:43:51.359350 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:43:51.359366 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:43:51.359797 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:43:51.360189 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:43:51.360194 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:43:51.360198 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:43:51.360388 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:43:51.360952 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:43:51.360962 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:43:51.361427 (+ 465us) client_negotiation.cc:769] Sending connection context
0925 10:43:51.361734 (+ 307us) client_negotiation.cc:240] Negotiation successful
0925 10:43:51.362281 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":343,"threads_started":1}
I20260925 10:43:51.365396 8837 heartbeater.cc:347] Connected to a master server at 127.23.100.62:45657
I20260925 10:43:51.365746 8837 heartbeater.cc:464] Registering TS with master...
I20260925 10:43:51.366576 8837 heartbeater.cc:511] Master 127.23.100.62:45657 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.100.2@KRBTEST.COM; defaulting to no policy
I20260925 10:43:51.368785 8648 ts_manager.cc:194] Registered new tserver with Master: 5eeeb702631649a8ae5270adcd356e87 (127.23.100.1:46059)
I20260925 10:43:51.370374 8648 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.1@KRBTEST.COM'} at 127.23.100.1:60763
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab.
Entry for principal kudu/127.23.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab.
I20260925 10:43:51.400920 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.2:0
--local_ip_for_outbound_sockets=127.23.100.2
--webserver_interface=127.23.100.2
--webserver_port=0
--tserver_master_addrs=127.23.100.62:45657
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab
--principal=kudu/127.23.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:37745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:43:51.725893 8843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:43:51.726224 8843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:43:51.726413 8843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:43:51.733518 8843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:43:51.733665 8843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:43:51.733848 8843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.2
I20260925 10:43:51.743403 8843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:37745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab
--principal=kudu/127.23.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.2
--webserver_port=0
--tserver_master_addrs=127.23.100.62:45657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.2
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:43:51.744660 8843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:43:51.745944 8843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:43:51.758718 8851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:43:51.758816 8848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:43:51.758795 8849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:43:51.759581 8843 server_base.cc:1034] running on GCE node
I20260925 10:43:51.760169 8843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:43:51.761060 8843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:43:51.762295 8843 hybrid_clock.cc:648] HybridClock initialized: now 1790333031762253 us; error 36 us; skew 500 ppm
I20260925 10:43:51.762667 8843 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:43:51 dist-test-slave-dxdd krb5kdc[8603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333031, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:43:51.765708 8843 init.cc:377] Logged in from keytab as kudu/127.23.100.2@KRBTEST.COM (short username kudu)
I20260925 10:43:51.767184 8843 webserver.cc:460] Webserver started at http://127.23.100.2:39855/ using document root <none> and password file <none>
I20260925 10:43:51.767853 8843 fs_manager.cc:359] Metadata directory not provided
I20260925 10:43:51.767998 8843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:43:51.768317 8843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:43:51.771196 8843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/ts-1/data/instance:
uuid: "6c385ca2ce834ac7b365a20204d988fd"
format_stamp: "Formatted at 2026-09-25 10:43:51 on dist-test-slave-dxdd"
server_key: "33c165aa0abdc8ecd5fa9672e67195d8"
server_key_iv: "c54f4a0af5fb2db33b8ff4d3fd7efcb3"
server_key_version: "encryptionkey@0"
I20260925 10:43:51.771958 8843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/ts-1/wal/instance:
uuid: "6c385ca2ce834ac7b365a20204d988fd"
format_stamp: "Formatted at 2026-09-25 10:43:51 on dist-test-slave-dxdd"
server_key: "33c165aa0abdc8ecd5fa9672e67195d8"
server_key_iv: "c54f4a0af5fb2db33b8ff4d3fd7efcb3"
server_key_version: "encryptionkey@0"
I20260925 10:43:51.777194 8843 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 10:43:51.780568 8858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:43:51.781828 8843 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:43:51.781996 8843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/ts-1/wal
uuid: "6c385ca2ce834ac7b365a20204d988fd"
format_stamp: "Formatted at 2026-09-25 10:43:51 on dist-test-slave-dxdd"
server_key: "33c165aa0abdc8ecd5fa9672e67195d8"
server_key_iv: "c54f4a0af5fb2db33b8ff4d3fd7efcb3"
server_key_version: "encryptionkey@0"
I20260925 10:43:51.782198 8843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:43:51.805146 8843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:43:51.809460 8843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:43:51.809746 8843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:43:51.810705 8843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:43:51.811957 8843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:43:51.812057 8843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:43:51.812203 8843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:43:51.812273 8843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:43:51.846084 8843 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.2:43467
I20260925 10:43:51.846174 8970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.2:43467 every 8 connection(s)
I20260925 10:43:51.847450 8843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
I20260925 10:43:51.854409 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 8843
I20260925 10:43:51.854612 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/ts-1/wal/instance
I20260925 10:43:51.855273 23952 external_mini_cluster.cc:1428] Setting key 19eb4f802097e2c6ffd0bc58cc5bbff2
Sep 25 10:43:51 dist-test-slave-dxdd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333031, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.2@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:43:51.869143 8839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:51.850069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:52031 (local address 127.23.100.62:45657)
0925 10:43:51.850267 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 10:43:51.850272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:43:51.851509 (+ 1237us) server_negotiation.cc:407] Connection header received
0925 10:43:51.853345 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:43:51.853350 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:43:51.853440 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:43:51.853560 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:43:51.855599 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:51.856299 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:51.858494 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:51.858906 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:51.859019 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:51.863279 (+ 4260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:43:51.863284 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:43:51.863288 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:43:51.863320 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:43:51.866099 (+ 2779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:43:51.866885 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:43:51.866890 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:43:51.866892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:43:51.867014 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:43:51.867442 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:43:51.867447 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:43:51.867449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:43:51.867698 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:43:51.867864 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 10:43:51.868539 (+ 675us) server_negotiation.cc:299] Negotiation successful
0925 10:43:51.868785 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 10:43:51.870128 8973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:51.850617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:45657 (local address 127.23.100.2:52031)
0925 10:43:51.851323 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:51.851367 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 10:43:51.853081 (+ 1714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:43:51.853775 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:43:51.853783 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:43:51.854574 (+ 791us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:43:51.855405 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:51.855445 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:51.856495 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:51.856499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:51.858350 (+ 1851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:51.858368 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:51.859154 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:51.859164 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:51.859489 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:51.860209 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:43:51.860243 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:43:51.863018 (+ 2775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:43:51.866255 (+ 3237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:43:51.866265 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:43:51.866281 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:43:51.866751 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:43:51.867129 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:43:51.867135 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:43:51.867139 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:43:51.867319 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:43:51.867838 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:43:51.867846 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:43:51.868179 (+ 333us) client_negotiation.cc:769] Sending connection context
0925 10:43:51.868440 (+ 261us) client_negotiation.cc:240] Negotiation successful
0925 10:43:51.868979 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":284,"threads_started":1}
I20260925 10:43:51.871694 8971 heartbeater.cc:347] Connected to a master server at 127.23.100.62:45657
I20260925 10:43:51.871989 8971 heartbeater.cc:464] Registering TS with master...
I20260925 10:43:51.872695 8971 heartbeater.cc:511] Master 127.23.100.62:45657 requested a full tablet report, sending...
I20260925 10:43:51.874182 8648 ts_manager.cc:194] Registered new tserver with Master: 6c385ca2ce834ac7b365a20204d988fd (127.23.100.2:43467)
I20260925 10:43:51.875026 8648 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.2@KRBTEST.COM'} at 127.23.100.2:52031
WARNING: no policy specified for kudu/127.23.100.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab.
Entry for principal kudu/127.23.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab.
I20260925 10:43:51.911504 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.3:0
--local_ip_for_outbound_sockets=127.23.100.3
--webserver_interface=127.23.100.3
--webserver_port=0
--tserver_master_addrs=127.23.100.62:45657
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab
--principal=kudu/127.23.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:37745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:43:52.233958 8976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:43:52.234200 8976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:43:52.234308 8976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:43:52.241303 8976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:43:52.241402 8976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:43:52.241519 8976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.3
I20260925 10:43:52.250733 8976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:37745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab
--principal=kudu/127.23.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.3
--webserver_port=0
--tserver_master_addrs=127.23.100.62:45657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.3
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:43:52.251848 8976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:43:52.253063 8976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:43:52.265516 8981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:43:52.265551 8984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:43:52.265762 8982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:43:52.266351 8976 server_base.cc:1034] running on GCE node
I20260925 10:43:52.266916 8976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:43:52.267766 8976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:43:52.268990 8976 hybrid_clock.cc:648] HybridClock initialized: now 1790333032268941 us; error 63 us; skew 500 ppm
I20260925 10:43:52.269281 8976 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:43:52 dist-test-slave-dxdd krb5kdc[8603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333032, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:43:52.272197 8976 init.cc:377] Logged in from keytab as kudu/127.23.100.3@KRBTEST.COM (short username kudu)
I20260925 10:43:52.273497 8976 webserver.cc:460] Webserver started at http://127.23.100.3:45379/ using document root <none> and password file <none>
I20260925 10:43:52.274195 8976 fs_manager.cc:359] Metadata directory not provided
I20260925 10:43:52.274384 8976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:43:52.274716 8976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:43:52.277518 8976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/ts-2/data/instance:
uuid: "84e9a94bfc054ebeaaf1a6eb413a6bf9"
format_stamp: "Formatted at 2026-09-25 10:43:52 on dist-test-slave-dxdd"
server_key: "23c01caffe9ecf7cad9cdcc8e7f17fbc"
server_key_iv: "a012b5160a6908050fa857fa11c0e0c3"
server_key_version: "encryptionkey@0"
I20260925 10:43:52.278184 8976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/ts-2/wal/instance:
uuid: "84e9a94bfc054ebeaaf1a6eb413a6bf9"
format_stamp: "Formatted at 2026-09-25 10:43:52 on dist-test-slave-dxdd"
server_key: "23c01caffe9ecf7cad9cdcc8e7f17fbc"
server_key_iv: "a012b5160a6908050fa857fa11c0e0c3"
server_key_version: "encryptionkey@0"
I20260925 10:43:52.283072 8976 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 10:43:52.286491 8991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:43:52.287700 8976 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 10:43:52.287869 8976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/ts-2/wal
uuid: "84e9a94bfc054ebeaaf1a6eb413a6bf9"
format_stamp: "Formatted at 2026-09-25 10:43:52 on dist-test-slave-dxdd"
server_key: "23c01caffe9ecf7cad9cdcc8e7f17fbc"
server_key_iv: "a012b5160a6908050fa857fa11c0e0c3"
server_key_version: "encryptionkey@0"
I20260925 10:43:52.288044 8976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:43:52.308333 8976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:43:52.312093 8976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:43:52.312363 8976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:43:52.313326 8976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:43:52.314656 8976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:43:52.314735 8976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:43:52.314901 8976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:43:52.314953 8976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:43:52.348143 8976 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.3:34829
I20260925 10:43:52.348258 9103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.3:34829 every 8 connection(s)
I20260925 10:43:52.349406 8976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
I20260925 10:43:52.355288 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 8976
I20260925 10:43:52.355480 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/ts-2/wal/instance
I20260925 10:43:52.355926 23952 external_mini_cluster.cc:1428] Setting key 09ea3685d4b4e55687b6f6e2cddb5596
Sep 25 10:43:52 dist-test-slave-dxdd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333032, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.3@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:43:52.368702 8839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:52.351873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:54775 (local address 127.23.100.62:45657)
0925 10:43:52.352088 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 10:43:52.352096 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:43:52.353267 (+ 1171us) server_negotiation.cc:407] Connection header received
0925 10:43:52.355240 (+ 1973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:43:52.355247 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:43:52.355337 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:43:52.355462 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:43:52.357121 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:52.358094 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:52.359711 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:52.360095 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:52.360198 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:52.363432 (+ 3234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:43:52.363437 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:43:52.363440 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:43:52.363471 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:43:52.366112 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:43:52.366809 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:43:52.366813 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:43:52.366815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:43:52.366938 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:43:52.367290 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:43:52.367293 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:43:52.367295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:43:52.367550 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:43:52.367659 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 10:43:52.368307 (+ 648us) server_negotiation.cc:299] Negotiation successful
0925 10:43:52.368464 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 10:43:52.369789 9106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:52.352415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:45657 (local address 127.23.100.3:54775)
0925 10:43:52.353108 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:52.353150 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 10:43:52.354981 (+ 1831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:43:52.355707 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:43:52.355722 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:43:52.356256 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:43:52.356912 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:52.356931 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:52.358243 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:52.358247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:52.359591 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:52.359602 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:52.360288 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:52.360293 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:52.360471 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:52.361046 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:43:52.361071 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:43:52.363254 (+ 2183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:43:52.366233 (+ 2979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:43:52.366239 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:43:52.366250 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:43:52.366693 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:43:52.367030 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:43:52.367033 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:43:52.367035 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:43:52.367195 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:43:52.367648 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:43:52.367653 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:43:52.367992 (+ 339us) client_negotiation.cc:769] Sending connection context
0925 10:43:52.368205 (+ 213us) client_negotiation.cc:240] Negotiation successful
0925 10:43:52.368560 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":262,"threads_started":1}
I20260925 10:43:52.371147 9104 heartbeater.cc:347] Connected to a master server at 127.23.100.62:45657
I20260925 10:43:52.371488 9104 heartbeater.cc:464] Registering TS with master...
I20260925 10:43:52.372151 9104 heartbeater.cc:511] Master 127.23.100.62:45657 requested a full tablet report, sending...
I20260925 10:43:52.373523 8648 ts_manager.cc:194] Registered new tserver with Master: 84e9a94bfc054ebeaaf1a6eb413a6bf9 (127.23.100.3:34829)
I20260925 10:43:52.373994 8837 heartbeater.cc:503] Master 127.23.100.62:45657 was elected leader, sending a full tablet report...
I20260925 10:43:52.374472 8648 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.3@KRBTEST.COM'} at 127.23.100.3:54775
I20260925 10:43:52.383955 23952 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:43:52.401698 8839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:52.388521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57664 (local address 127.23.100.62:45657)
0925 10:43:52.388714 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 10:43:52.388723 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:43:52.389170 (+ 447us) server_negotiation.cc:407] Connection header received
0925 10:43:52.389392 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:43:52.389397 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:43:52.389473 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:43:52.389575 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:43:52.391135 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:52.391850 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:52.393072 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:52.393519 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:52.393657 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:52.396307 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:43:52.396313 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:43:52.396316 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:43:52.396348 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:43:52.399039 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:43:52.399704 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:43:52.399712 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:43:52.399715 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:43:52.399849 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:43:52.400220 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:43:52.400226 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:43:52.400229 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:43:52.400560 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:43:52.400834 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0925 10:43:52.401168 (+ 334us) server_negotiation.cc:299] Negotiation successful
0925 10:43:52.401378 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 10:43:52.405947 8647 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57664:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:43:52.413297 8647 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:43:52.458832 9117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:52.424673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:46059 (local address 127.0.0.1:51092)
0925 10:43:52.450812 (+ 26139us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:52.450838 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 10:43:52.450997 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:43:52.452056 (+ 1059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:43:52.452064 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:43:52.452099 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:43:52.452525 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:52.452539 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:52.454394 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:52.454402 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:52.456421 (+ 2019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:52.456431 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:52.457713 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:52.457720 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:52.458033 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:52.458071 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 10:43:52.458222 (+ 151us) client_negotiation.cc:240] Negotiation successful
0925 10:43:52.458434 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25951,"mutex_wait_us":80,"thread_start_us":26999,"threads_started":1}
I20260925 10:43:52.459547 9120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:52.425200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51092 (local address 127.23.100.1:46059)
0925 10:43:52.450466 (+ 25266us) server_negotiation.cc:206] Beginning negotiation
0925 10:43:52.450479 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 10:43:52.451145 (+ 666us) server_negotiation.cc:407] Connection header received
0925 10:43:52.451278 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:43:52.451289 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:43:52.451593 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:43:52.451794 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:43:52.452720 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:52.454253 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:52.456565 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:52.457502 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:52.457656 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:52.457926 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0925 10:43:52.459079 (+ 1153us) server_negotiation.cc:299] Negotiation successful
0925 10:43:52.459250 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24994,"thread_start_us":24834,"threads_started":1}
I20260925 10:43:52.460703 9118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:52.424819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35194 (local address 127.23.100.3:34829)
0925 10:43:52.452513 (+ 27694us) server_negotiation.cc:206] Beginning negotiation
0925 10:43:52.452521 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:43:52.452551 (+ 30us) server_negotiation.cc:407] Connection header received
0925 10:43:52.452650 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:43:52.452660 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:43:52.452871 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:43:52.453058 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:43:52.453640 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:52.455115 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:52.457842 (+ 2727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:52.459314 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:52.459450 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:52.459730 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0925 10:43:52.460204 (+ 474us) server_negotiation.cc:299] Negotiation successful
0925 10:43:52.460376 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27526,"thread_start_us":27382,"threads_started":1}
I20260925 10:43:52.460709 9119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:52.424522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.3:34829 (local address 127.0.0.1:35194)
0925 10:43:52.449244 (+ 24722us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:52.449306 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 10:43:52.449570 (+ 264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:43:52.453051 (+ 3481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:43:52.453055 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:43:52.453078 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:43:52.453449 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:52.453459 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:52.455273 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:52.455278 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:52.457660 (+ 2382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:52.457674 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:52.459856 (+ 2182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:52.459862 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:52.460015 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:52.460043 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 10:43:52.460142 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 10:43:52.460272 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24326,"mutex_wait_us":1,"thread_start_us":26941,"threads_started":1}
I20260925 10:43:52.462416 9116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:52.425090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.2:43467 (local address 127.0.0.1:57356)
0925 10:43:52.451831 (+ 26741us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:52.451853 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 10:43:52.452595 (+ 742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:43:52.453499 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:43:52.453503 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:43:52.453527 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:43:52.453869 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:52.453880 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:52.455737 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:52.455741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:52.458041 (+ 2300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:52.458055 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:52.461658 (+ 3603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:52.461664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:52.461819 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:52.461848 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 10:43:52.461942 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 10:43:52.462055 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26555,"mutex_wait_us":57,"thread_start_us":25428,"threads_started":1}
I20260925 10:43:52.462661 9121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:52.425682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57356 (local address 127.23.100.2:43467)
0925 10:43:52.448204 (+ 22522us) server_negotiation.cc:206] Beginning negotiation
0925 10:43:52.448218 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 10:43:52.451979 (+ 3761us) server_negotiation.cc:407] Connection header received
0925 10:43:52.452790 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:43:52.452798 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:43:52.453034 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:43:52.453193 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:43:52.454055 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:52.455572 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:52.458192 (+ 2620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:52.461432 (+ 3240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:52.461581 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:52.461782 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 10:43:52.461995 (+ 213us) server_negotiation.cc:299] Negotiation successful
0925 10:43:52.462153 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22281,"thread_start_us":22551,"threads_started":1}
I20260925 10:43:52.463629 8772 tablet_service.cc:1467] Processing CreateTablet for tablet f61083f48e2d4e6bbebeee091348beb6 (DEFAULT_TABLE table=test-table [id=41a5f154439f4c349204121653504362]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:43:52.464046 9039 tablet_service.cc:1467] Processing CreateTablet for tablet f61083f48e2d4e6bbebeee091348beb6 (DEFAULT_TABLE table=test-table [id=41a5f154439f4c349204121653504362]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:43:52.465091 8772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f61083f48e2d4e6bbebeee091348beb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:43:52.465200 9039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f61083f48e2d4e6bbebeee091348beb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:43:52.465813 8906 tablet_service.cc:1467] Processing CreateTablet for tablet f61083f48e2d4e6bbebeee091348beb6 (DEFAULT_TABLE table=test-table [id=41a5f154439f4c349204121653504362]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:43:52.467099 8906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f61083f48e2d4e6bbebeee091348beb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:43:52.475958 9122 tablet_bootstrap.cc:492] T f61083f48e2d4e6bbebeee091348beb6 P 84e9a94bfc054ebeaaf1a6eb413a6bf9: Bootstrap starting.
I20260925 10:43:52.478956 9122 tablet_bootstrap.cc:654] T f61083f48e2d4e6bbebeee091348beb6 P 84e9a94bfc054ebeaaf1a6eb413a6bf9: Neither blocks nor log segments found. Creating new log.
I20260925 10:43:52.479938 9122 log.cc:824] T f61083f48e2d4e6bbebeee091348beb6 P 84e9a94bfc054ebeaaf1a6eb413a6bf9: Log is configured to *not* fsync() on all Append() calls
I20260925 10:43:52.480816 9124 tablet_bootstrap.cc:492] T f61083f48e2d4e6bbebeee091348beb6 P 5eeeb702631649a8ae5270adcd356e87: Bootstrap starting.
I20260925 10:43:52.481724 9123 tablet_bootstrap.cc:492] T f61083f48e2d4e6bbebeee091348beb6 P 6c385ca2ce834ac7b365a20204d988fd: Bootstrap starting.
I20260925 10:43:52.485608 9124 tablet_bootstrap.cc:654] T f61083f48e2d4e6bbebeee091348beb6 P 5eeeb702631649a8ae5270adcd356e87: Neither blocks nor log segments found. Creating new log.
I20260925 10:43:52.486040 9123 tablet_bootstrap.cc:654] T f61083f48e2d4e6bbebeee091348beb6 P 6c385ca2ce834ac7b365a20204d988fd: Neither blocks nor log segments found. Creating new log.
I20260925 10:43:52.486955 9124 log.cc:824] T f61083f48e2d4e6bbebeee091348beb6 P 5eeeb702631649a8ae5270adcd356e87: Log is configured to *not* fsync() on all Append() calls
I20260925 10:43:52.487200 9123 log.cc:824] T f61083f48e2d4e6bbebeee091348beb6 P 6c385ca2ce834ac7b365a20204d988fd: Log is configured to *not* fsync() on all Append() calls
I20260925 10:43:52.488306 9122 tablet_bootstrap.cc:492] T f61083f48e2d4e6bbebeee091348beb6 P 84e9a94bfc054ebeaaf1a6eb413a6bf9: No bootstrap required, opened a new log
I20260925 10:43:52.488613 9122 ts_tablet_manager.cc:1397] T f61083f48e2d4e6bbebeee091348beb6 P 84e9a94bfc054ebeaaf1a6eb413a6bf9: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20260925 10:43:52.489930 9124 tablet_bootstrap.cc:492] T f61083f48e2d4e6bbebeee091348beb6 P 5eeeb702631649a8ae5270adcd356e87: No bootstrap required, opened a new log
I20260925 10:43:52.489941 9123 tablet_bootstrap.cc:492] T f61083f48e2d4e6bbebeee091348beb6 P 6c385ca2ce834ac7b365a20204d988fd: No bootstrap required, opened a new log
I20260925 10:43:52.490180 9124 ts_tablet_manager.cc:1397] T f61083f48e2d4e6bbebeee091348beb6 P 5eeeb702631649a8ae5270adcd356e87: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260925 10:43:52.490171 9123 ts_tablet_manager.cc:1397] T f61083f48e2d4e6bbebeee091348beb6 P 6c385ca2ce834ac7b365a20204d988fd: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260925 10:43:52.497010 9122 raft_consensus.cc:348] T f61083f48e2d4e6bbebeee091348beb6 P 84e9a94bfc054ebeaaf1a6eb413a6bf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84e9a94bfc054ebeaaf1a6eb413a6bf9" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 34829 } } peers { permanent_uuid: "5eeeb702631649a8ae5270adcd356e87" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 46059 } } peers { permanent_uuid: "6c385ca2ce834ac7b365a20204d988fd" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 43467 } }
I20260925 10:43:52.497370 9122 raft_consensus.cc:374] T f61083f48e2d4e6bbebeee091348beb6 P 84e9a94bfc054ebeaaf1a6eb413a6bf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:43:52.497466 9122 raft_consensus.cc:729] T f61083f48e2d4e6bbebeee091348beb6 P 84e9a94bfc054ebeaaf1a6eb413a6bf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84e9a94bfc054ebeaaf1a6eb413a6bf9, State: Initialized, Role: FOLLOWER
I20260925 10:43:52.498201 9122 consensus_queue.cc:260] T f61083f48e2d4e6bbebeee091348beb6 P 84e9a94bfc054ebeaaf1a6eb413a6bf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84e9a94bfc054ebeaaf1a6eb413a6bf9" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 34829 } } peers { permanent_uuid: "5eeeb702631649a8ae5270adcd356e87" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 46059 } } peers { permanent_uuid: "6c385ca2ce834ac7b365a20204d988fd" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 43467 } }
I20260925 10:43:52.500785 9123 raft_consensus.cc:348] T f61083f48e2d4e6bbebeee091348beb6 P 6c385ca2ce834ac7b365a20204d988fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84e9a94bfc054ebeaaf1a6eb413a6bf9" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 34829 } } peers { permanent_uuid: "5eeeb702631649a8ae5270adcd356e87" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 46059 } } peers { permanent_uuid: "6c385ca2ce834ac7b365a20204d988fd" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 43467 } }
I20260925 10:43:52.500785 9124 raft_consensus.cc:348] T f61083f48e2d4e6bbebeee091348beb6 P 5eeeb702631649a8ae5270adcd356e87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84e9a94bfc054ebeaaf1a6eb413a6bf9" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 34829 } } peers { permanent_uuid: "5eeeb702631649a8ae5270adcd356e87" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 46059 } } peers { permanent_uuid: "6c385ca2ce834ac7b365a20204d988fd" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 43467 } }
I20260925 10:43:52.501220 9124 raft_consensus.cc:374] T f61083f48e2d4e6bbebeee091348beb6 P 5eeeb702631649a8ae5270adcd356e87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:43:52.501235 9123 raft_consensus.cc:374] T f61083f48e2d4e6bbebeee091348beb6 P 6c385ca2ce834ac7b365a20204d988fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:43:52.501333 9123 raft_consensus.cc:729] T f61083f48e2d4e6bbebeee091348beb6 P 6c385ca2ce834ac7b365a20204d988fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c385ca2ce834ac7b365a20204d988fd, State: Initialized, Role: FOLLOWER
I20260925 10:43:52.501333 9124 raft_consensus.cc:729] T f61083f48e2d4e6bbebeee091348beb6 P 5eeeb702631649a8ae5270adcd356e87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5eeeb702631649a8ae5270adcd356e87, State: Initialized, Role: FOLLOWER
I20260925 10:43:52.501947 9123 consensus_queue.cc:260] T f61083f48e2d4e6bbebeee091348beb6 P 6c385ca2ce834ac7b365a20204d988fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84e9a94bfc054ebeaaf1a6eb413a6bf9" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 34829 } } peers { permanent_uuid: "5eeeb702631649a8ae5270adcd356e87" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 46059 } } peers { permanent_uuid: "6c385ca2ce834ac7b365a20204d988fd" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 43467 } }
I20260925 10:43:52.501976 9124 consensus_queue.cc:260] T f61083f48e2d4e6bbebeee091348beb6 P 5eeeb702631649a8ae5270adcd356e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84e9a94bfc054ebeaaf1a6eb413a6bf9" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 34829 } } peers { permanent_uuid: "5eeeb702631649a8ae5270adcd356e87" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 46059 } } peers { permanent_uuid: "6c385ca2ce834ac7b365a20204d988fd" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 43467 } }
I20260925 10:43:52.502162 9104 heartbeater.cc:503] Master 127.23.100.62:45657 was elected leader, sending a full tablet report...
I20260925 10:43:52.503541 8971 heartbeater.cc:503] Master 127.23.100.62:45657 was elected leader, sending a full tablet report...
I20260925 10:43:52.503845 9123 ts_tablet_manager.cc:1428] T f61083f48e2d4e6bbebeee091348beb6 P 6c385ca2ce834ac7b365a20204d988fd: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260925 10:43:52.504729 9122 ts_tablet_manager.cc:1428] T f61083f48e2d4e6bbebeee091348beb6 P 84e9a94bfc054ebeaaf1a6eb413a6bf9: Time spent starting tablet: real 0.016s user 0.008s sys 0.005s
I20260925 10:43:52.505767 9124 ts_tablet_manager.cc:1428] T f61083f48e2d4e6bbebeee091348beb6 P 5eeeb702631649a8ae5270adcd356e87: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
W20260925 10:43:52.593132 8838 tablet.cc:2369] T f61083f48e2d4e6bbebeee091348beb6 P 5eeeb702631649a8ae5270adcd356e87: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:43:52.599380 8972 tablet.cc:2369] T f61083f48e2d4e6bbebeee091348beb6 P 6c385ca2ce834ac7b365a20204d988fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:43:52.600976 9105 tablet.cc:2369] T f61083f48e2d4e6bbebeee091348beb6 P 84e9a94bfc054ebeaaf1a6eb413a6bf9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:43:52.623697 9129 raft_consensus.cc:482] T f61083f48e2d4e6bbebeee091348beb6 P 6c385ca2ce834ac7b365a20204d988fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:43:52.623991 9129 raft_consensus.cc:504] T f61083f48e2d4e6bbebeee091348beb6 P 6c385ca2ce834ac7b365a20204d988fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84e9a94bfc054ebeaaf1a6eb413a6bf9" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 34829 } } peers { permanent_uuid: "5eeeb702631649a8ae5270adcd356e87" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 46059 } } peers { permanent_uuid: "6c385ca2ce834ac7b365a20204d988fd" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 43467 } }
I20260925 10:43:52.625521 9129 leader_election.cc:288] T f61083f48e2d4e6bbebeee091348beb6 P 6c385ca2ce834ac7b365a20204d988fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84e9a94bfc054ebeaaf1a6eb413a6bf9 (127.23.100.3:34829), 5eeeb702631649a8ae5270adcd356e87 (127.23.100.1:46059)
I20260925 10:43:52.632122 9120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:52.626010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:52089 (local address 127.23.100.1:46059)
0925 10:43:52.626236 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 10:43:52.626245 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:43:52.626962 (+ 717us) server_negotiation.cc:407] Connection header received
0925 10:43:52.627185 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:43:52.627189 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:43:52.627249 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:43:52.627351 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:43:52.627904 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:52.628616 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:52.630452 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:52.631210 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:52.631398 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:52.631474 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 10:43:52.631764 (+ 290us) server_negotiation.cc:299] Negotiation successful
0925 10:43:52.631886 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 10:43:52.632205 9131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:52.626005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:46059 (local address 127.23.100.2:52089)
0925 10:43:52.626832 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:52.626847 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:43:52.627025 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:43:52.627377 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:43:52.627381 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:43:52.627405 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:43:52.627762 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:52.627770 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:52.628751 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:52.628758 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:52.630255 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:52.630267 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:52.631419 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:52.631425 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:52.631586 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:52.631609 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 10:43:52.631721 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 10:43:52.631864 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"thread_start_us":485,"threads_started":1}
I20260925 10:43:52.632328 9118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:52.626047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:58887 (local address 127.23.100.3:34829)
0925 10:43:52.626268 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 10:43:52.626276 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:43:52.627095 (+ 819us) server_negotiation.cc:407] Connection header received
0925 10:43:52.627314 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:43:52.627318 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:43:52.627375 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:43:52.627472 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:43:52.628102 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:52.628818 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:52.630609 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:52.631366 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:52.631542 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:52.631617 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 10:43:52.631937 (+ 320us) server_negotiation.cc:299] Negotiation successful
0925 10:43:52.632056 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 10:43:52.632617 9132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:52.626116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.3:34829 (local address 127.23.100.2:58887)
0925 10:43:52.626979 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:52.626993 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:43:52.627155 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:43:52.627495 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:43:52.627499 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:43:52.627538 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:43:52.627945 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:52.627953 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:52.628967 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:52.628973 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:52.630457 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:52.630469 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:52.631581 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:52.631588 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:52.631787 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:52.631808 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 10:43:52.631908 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 10:43:52.632029 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"thread_start_us":530,"threads_started":1}
I20260925 10:43:52.633221 8792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61083f48e2d4e6bbebeee091348beb6" candidate_uuid: "6c385ca2ce834ac7b365a20204d988fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5eeeb702631649a8ae5270adcd356e87" is_pre_election: true
I20260925 10:43:52.633539 9059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61083f48e2d4e6bbebeee091348beb6" candidate_uuid: "6c385ca2ce834ac7b365a20204d988fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84e9a94bfc054ebeaaf1a6eb413a6bf9" is_pre_election: true
I20260925 10:43:52.633731 8792 raft_consensus.cc:2454] T f61083f48e2d4e6bbebeee091348beb6 P 5eeeb702631649a8ae5270adcd356e87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c385ca2ce834ac7b365a20204d988fd in term 0.
I20260925 10:43:52.634040 9059 raft_consensus.cc:2454] T f61083f48e2d4e6bbebeee091348beb6 P 84e9a94bfc054ebeaaf1a6eb413a6bf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c385ca2ce834ac7b365a20204d988fd in term 0.
I20260925 10:43:52.634476 8862 leader_election.cc:302] T f61083f48e2d4e6bbebeee091348beb6 P 6c385ca2ce834ac7b365a20204d988fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5eeeb702631649a8ae5270adcd356e87, 6c385ca2ce834ac7b365a20204d988fd; no voters:
I20260925 10:43:52.634920 9129 raft_consensus.cc:2781] T f61083f48e2d4e6bbebeee091348beb6 P 6c385ca2ce834ac7b365a20204d988fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:43:52.635097 9129 raft_consensus.cc:482] T f61083f48e2d4e6bbebeee091348beb6 P 6c385ca2ce834ac7b365a20204d988fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:43:52.635221 9129 raft_consensus.cc:3037] T f61083f48e2d4e6bbebeee091348beb6 P 6c385ca2ce834ac7b365a20204d988fd [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:43:52.637344 9129 raft_consensus.cc:504] T f61083f48e2d4e6bbebeee091348beb6 P 6c385ca2ce834ac7b365a20204d988fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84e9a94bfc054ebeaaf1a6eb413a6bf9" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 34829 } } peers { permanent_uuid: "5eeeb702631649a8ae5270adcd356e87" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 46059 } } peers { permanent_uuid: "6c385ca2ce834ac7b365a20204d988fd" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 43467 } }
I20260925 10:43:52.637914 9129 leader_election.cc:288] T f61083f48e2d4e6bbebeee091348beb6 P 6c385ca2ce834ac7b365a20204d988fd [CANDIDATE]: Term 1 election: Requested vote from peers 84e9a94bfc054ebeaaf1a6eb413a6bf9 (127.23.100.3:34829), 5eeeb702631649a8ae5270adcd356e87 (127.23.100.1:46059)
I20260925 10:43:52.638438 9059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61083f48e2d4e6bbebeee091348beb6" candidate_uuid: "6c385ca2ce834ac7b365a20204d988fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84e9a94bfc054ebeaaf1a6eb413a6bf9"
I20260925 10:43:52.638446 8792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61083f48e2d4e6bbebeee091348beb6" candidate_uuid: "6c385ca2ce834ac7b365a20204d988fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5eeeb702631649a8ae5270adcd356e87"
I20260925 10:43:52.638636 9059 raft_consensus.cc:3037] T f61083f48e2d4e6bbebeee091348beb6 P 84e9a94bfc054ebeaaf1a6eb413a6bf9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:43:52.638660 8792 raft_consensus.cc:3037] T f61083f48e2d4e6bbebeee091348beb6 P 5eeeb702631649a8ae5270adcd356e87 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:43:52.640784 9059 raft_consensus.cc:2454] T f61083f48e2d4e6bbebeee091348beb6 P 84e9a94bfc054ebeaaf1a6eb413a6bf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c385ca2ce834ac7b365a20204d988fd in term 1.
I20260925 10:43:52.640807 8792 raft_consensus.cc:2454] T f61083f48e2d4e6bbebeee091348beb6 P 5eeeb702631649a8ae5270adcd356e87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c385ca2ce834ac7b365a20204d988fd in term 1.
I20260925 10:43:52.641176 8860 leader_election.cc:302] T f61083f48e2d4e6bbebeee091348beb6 P 6c385ca2ce834ac7b365a20204d988fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c385ca2ce834ac7b365a20204d988fd, 84e9a94bfc054ebeaaf1a6eb413a6bf9; no voters:
I20260925 10:43:52.641458 9129 raft_consensus.cc:2781] T f61083f48e2d4e6bbebeee091348beb6 P 6c385ca2ce834ac7b365a20204d988fd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:43:52.641901 9129 raft_consensus.cc:686] T f61083f48e2d4e6bbebeee091348beb6 P 6c385ca2ce834ac7b365a20204d988fd [term 1 LEADER]: Becoming Leader. State: Replica: 6c385ca2ce834ac7b365a20204d988fd, State: Running, Role: LEADER
I20260925 10:43:52.642400 9129 consensus_queue.cc:237] T f61083f48e2d4e6bbebeee091348beb6 P 6c385ca2ce834ac7b365a20204d988fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84e9a94bfc054ebeaaf1a6eb413a6bf9" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 34829 } } peers { permanent_uuid: "5eeeb702631649a8ae5270adcd356e87" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 46059 } } peers { permanent_uuid: "6c385ca2ce834ac7b365a20204d988fd" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 43467 } }
I20260925 10:43:52.647090 8648 catalog_manager.cc:5495] T f61083f48e2d4e6bbebeee091348beb6 P 6c385ca2ce834ac7b365a20204d988fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c385ca2ce834ac7b365a20204d988fd (127.23.100.2). New cstate: current_term: 1 leader_uuid: "6c385ca2ce834ac7b365a20204d988fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84e9a94bfc054ebeaaf1a6eb413a6bf9" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 34829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5eeeb702631649a8ae5270adcd356e87" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 46059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c385ca2ce834ac7b365a20204d988fd" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 43467 } health_report { overall_health: HEALTHY } } }
I20260925 10:43:52.707479 9121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:52.702186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57366 (local address 127.23.100.2:43467)
0925 10:43:52.702406 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 10:43:52.702412 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:43:52.702479 (+ 67us) server_negotiation.cc:407] Connection header received
0925 10:43:52.702725 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:43:52.702730 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:43:52.702824 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:43:52.702971 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:43:52.703350 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:52.704072 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:52.705430 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:52.705809 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:52.705953 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:52.706461 (+ 508us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:43:52.706926 (+ 465us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:43:52.707082 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 10:43:52.707135 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 10:43:52.707221 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 10:43:52.715008 8792 raft_consensus.cc:1261] T f61083f48e2d4e6bbebeee091348beb6 P 5eeeb702631649a8ae5270adcd356e87 [term 1 FOLLOWER]: Refusing update from remote peer 6c385ca2ce834ac7b365a20204d988fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:43:52.715058 9059 raft_consensus.cc:1261] T f61083f48e2d4e6bbebeee091348beb6 P 84e9a94bfc054ebeaaf1a6eb413a6bf9 [term 1 FOLLOWER]: Refusing update from remote peer 6c385ca2ce834ac7b365a20204d988fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:43:52.715963 9133 consensus_queue.cc:1035] T f61083f48e2d4e6bbebeee091348beb6 P 6c385ca2ce834ac7b365a20204d988fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "84e9a94bfc054ebeaaf1a6eb413a6bf9" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 34829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:43:52.716271 9129 consensus_queue.cc:1035] T f61083f48e2d4e6bbebeee091348beb6 P 6c385ca2ce834ac7b365a20204d988fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eeeb702631649a8ae5270adcd356e87" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 46059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:43:52.746116 8647 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:43:52.746479 8647 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:43:52.750533 8647 catalog_manager.cc:5782] T 00000000000000000000000000000000 P acfdd5b38c3f4aa8af5afd47c8de3226: Sending DeleteTablet for 3 replicas of tablet f61083f48e2d4e6bbebeee091348beb6
I20260925 10:43:52.751646 9039 tablet_service.cc:1514] Processing DeleteTablet for tablet f61083f48e2d4e6bbebeee091348beb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:43:52 UTC) from {username='kudu', principal='kudu/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:35194
I20260925 10:43:52.751956 8906 tablet_service.cc:1514] Processing DeleteTablet for tablet f61083f48e2d4e6bbebeee091348beb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:43:52 UTC) from {username='kudu', principal='kudu/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:57356
I20260925 10:43:52.751950 8772 tablet_service.cc:1514] Processing DeleteTablet for tablet f61083f48e2d4e6bbebeee091348beb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:43:52 UTC) from {username='kudu', principal='kudu/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:51092
I20260925 10:43:52.753644 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 8709
I20260925 10:43:52.764250 9146 tablet_replica.cc:330] T f61083f48e2d4e6bbebeee091348beb6 P 84e9a94bfc054ebeaaf1a6eb413a6bf9: stopping tablet replica
I20260925 10:43:52.764823 9146 raft_consensus.cc:2229] T f61083f48e2d4e6bbebeee091348beb6 P 84e9a94bfc054ebeaaf1a6eb413a6bf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:43:52.765450 9146 raft_consensus.cc:2258] T f61083f48e2d4e6bbebeee091348beb6 P 84e9a94bfc054ebeaaf1a6eb413a6bf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:43:52.767314 9148 tablet_replica.cc:330] T f61083f48e2d4e6bbebeee091348beb6 P 5eeeb702631649a8ae5270adcd356e87: stopping tablet replica
I20260925 10:43:52.767680 9147 tablet_replica.cc:330] T f61083f48e2d4e6bbebeee091348beb6 P 6c385ca2ce834ac7b365a20204d988fd: stopping tablet replica
I20260925 10:43:52.767750 9148 raft_consensus.cc:2229] T f61083f48e2d4e6bbebeee091348beb6 P 5eeeb702631649a8ae5270adcd356e87 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:43:52.768082 9147 raft_consensus.cc:2229] T f61083f48e2d4e6bbebeee091348beb6 P 6c385ca2ce834ac7b365a20204d988fd [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:43:52.768247 9148 raft_consensus.cc:2258] T f61083f48e2d4e6bbebeee091348beb6 P 5eeeb702631649a8ae5270adcd356e87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:43:52.769245 9147 raft_consensus.cc:2258] T f61083f48e2d4e6bbebeee091348beb6 P 6c385ca2ce834ac7b365a20204d988fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:43:52.770929 9148 ts_tablet_manager.cc:1905] T f61083f48e2d4e6bbebeee091348beb6 P 5eeeb702631649a8ae5270adcd356e87: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:43:52.770933 9146 ts_tablet_manager.cc:1905] T f61083f48e2d4e6bbebeee091348beb6 P 84e9a94bfc054ebeaaf1a6eb413a6bf9: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 25 10:43:52 dist-test-slave-dxdd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.1@KRBTEST.COM
I20260925 10:43:52.777544 9147 ts_tablet_manager.cc:1905] T f61083f48e2d4e6bbebeee091348beb6 P 6c385ca2ce834ac7b365a20204d988fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:43:52.777978 9148 ts_tablet_manager.cc:1918] T f61083f48e2d4e6bbebeee091348beb6 P 5eeeb702631649a8ae5270adcd356e87: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:43:52.778152 9148 log.cc:1196] T f61083f48e2d4e6bbebeee091348beb6 P 5eeeb702631649a8ae5270adcd356e87: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/ts-0/wal/wals/f61083f48e2d4e6bbebeee091348beb6
I20260925 10:43:52.778684 9148 ts_tablet_manager.cc:1939] T f61083f48e2d4e6bbebeee091348beb6 P 5eeeb702631649a8ae5270adcd356e87: Deleting consensus metadata
I20260925 10:43:52.780637 8635 catalog_manager.cc:4841] TS 5eeeb702631649a8ae5270adcd356e87 (127.23.100.1:46059): tablet f61083f48e2d4e6bbebeee091348beb6 (table test-table [id=41a5f154439f4c349204121653504362]) successfully deleted
I20260925 10:43:52.783496 9146 ts_tablet_manager.cc:1918] T f61083f48e2d4e6bbebeee091348beb6 P 84e9a94bfc054ebeaaf1a6eb413a6bf9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:43:52.783679 9146 log.cc:1196] T f61083f48e2d4e6bbebeee091348beb6 P 84e9a94bfc054ebeaaf1a6eb413a6bf9: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/ts-2/wal/wals/f61083f48e2d4e6bbebeee091348beb6
I20260925 10:43:52.784186 9146 ts_tablet_manager.cc:1939] T f61083f48e2d4e6bbebeee091348beb6 P 84e9a94bfc054ebeaaf1a6eb413a6bf9: Deleting consensus metadata
I20260925 10:43:52.784756 9120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:52.755122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51108 (local address 127.23.100.1:46059)
0925 10:43:52.755300 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 10:43:52.755307 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:43:52.760971 (+ 5664us) server_negotiation.cc:407] Connection header received
0925 10:43:52.761190 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:43:52.761196 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:43:52.761284 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:43:52.761392 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:43:52.766013 (+ 4621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:52.767292 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:52.769637 (+ 2345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:52.770291 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:52.770429 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:52.776197 (+ 5768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:43:52.776212 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:43:52.776229 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:43:52.776282 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:43:52.780586 (+ 4304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:43:52.781682 (+ 1096us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:43:52.781691 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:43:52.781695 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:43:52.781856 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:43:52.782397 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:43:52.782402 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:43:52.782405 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:43:52.782957 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:43:52.783232 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0925 10:43:52.784062 (+ 830us) server_negotiation.cc:299] Negotiation successful
0925 10:43:52.784363 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 10:43:52.784876 9147 ts_tablet_manager.cc:1918] T f61083f48e2d4e6bbebeee091348beb6 P 6c385ca2ce834ac7b365a20204d988fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:43:52.785053 9147 log.cc:1196] T f61083f48e2d4e6bbebeee091348beb6 P 6c385ca2ce834ac7b365a20204d988fd: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790332740062128-23952-0/minicluster-data/ts-1/wal/wals/f61083f48e2d4e6bbebeee091348beb6
I20260925 10:43:52.785764 9147 ts_tablet_manager.cc:1939] T f61083f48e2d4e6bbebeee091348beb6 P 6c385ca2ce834ac7b365a20204d988fd: Deleting consensus metadata
I20260925 10:43:52.786031 8633 catalog_manager.cc:4841] TS 84e9a94bfc054ebeaaf1a6eb413a6bf9 (127.23.100.3:34829): tablet f61083f48e2d4e6bbebeee091348beb6 (table test-table [id=41a5f154439f4c349204121653504362]) successfully deleted
I20260925 10:43:52.786899 8832 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:43:52.787793 8632 catalog_manager.cc:4841] TS 6c385ca2ce834ac7b365a20204d988fd (127.23.100.2:43467): tablet f61083f48e2d4e6bbebeee091348beb6 (table test-table [id=41a5f154439f4c349204121653504362]) successfully deleted
I20260925 10:43:52.876732 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 8709
I20260925 10:43:52.895206 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 8843
Sep 25 10:43:52 dist-test-slave-dxdd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.2@KRBTEST.COM
I20260925 10:43:52.912415 9121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:52.896151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57378 (local address 127.23.100.2:43467)
0925 10:43:52.896340 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 10:43:52.896345 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:43:52.896362 (+ 17us) server_negotiation.cc:407] Connection header received
0925 10:43:52.896534 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:43:52.896538 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:43:52.896600 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:43:52.896688 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:43:52.898463 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:52.899305 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:52.901205 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:52.901631 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:52.901721 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:52.906567 (+ 4846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:43:52.906577 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:43:52.906591 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:43:52.906629 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:43:52.909553 (+ 2924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:43:52.910251 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:43:52.910262 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:43:52.910270 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:43:52.910466 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:43:52.910848 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:43:52.910854 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:43:52.910857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:43:52.911215 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:43:52.911536 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0925 10:43:52.911851 (+ 315us) server_negotiation.cc:299] Negotiation successful
0925 10:43:52.912056 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 10:43:52.913177 8966 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:43:53.025935 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 8843
I20260925 10:43:53.044480 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 8976
Sep 25 10:43:53 dist-test-slave-dxdd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.3@KRBTEST.COM
I20260925 10:43:53.061205 9118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:53.045287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35208 (local address 127.23.100.3:34829)
0925 10:43:53.045475 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 10:43:53.045479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:43:53.045593 (+ 114us) server_negotiation.cc:407] Connection header received
0925 10:43:53.045769 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:43:53.045772 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:43:53.045836 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:43:53.045961 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:43:53.047362 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:53.048151 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:53.049398 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:53.049797 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:53.049928 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:53.055982 (+ 6054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:43:53.056017 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:43:53.056032 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:43:53.056067 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:43:53.058743 (+ 2676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:43:53.059401 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:43:53.059408 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:43:53.059410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:43:53.059546 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:43:53.059878 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:43:53.059881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:43:53.059883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:43:53.060192 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:43:53.060405 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0925 10:43:53.060649 (+ 244us) server_negotiation.cc:299] Negotiation successful
0925 10:43:53.060807 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 10:43:53.061860 9099 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:43:53.148487 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 8976
I20260925 10:43:53.167486 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 8616
I20260925 10:43:53.168644 8678 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:43:53.277905 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 8616
2026-09-25T10:43:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4276 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T10:43:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:43:53Z Disabled control of system clock
I20260925 10:43:53.352293 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:35089
--webserver_interface=127.23.100.62
--webserver_port=0
--builtin_ntp_servers=127.23.100.20:35443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.62:35089
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 10:43:53.670416 9157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:43:53.670702 9157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:43:53.670835 9157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:43:53.677644 9157 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:43:53.677755 9157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:43:53.677819 9157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:43:53.677913 9157 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:43:53.677960 9157 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:43:53.686642 9157 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:35443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.62:35089
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:35089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:43:53.687808 9157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:43:53.689524 9157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:43:53.701059 9163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:43:53.701081 9162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:43:53.701143 9165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:43:53.701781 9157 server_base.cc:1034] running on GCE node
I20260925 10:43:53.702546 9157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:43:53.703763 9157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:43:53.704959 9157 hybrid_clock.cc:648] HybridClock initialized: now 1790333033704920 us; error 38 us; skew 500 ppm
I20260925 10:43:53.705281 9157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:43:53.707336 9157 webserver.cc:460] Webserver started at http://127.23.100.62:46559/ using document root <none> and password file <none>
I20260925 10:43:53.707959 9157 fs_manager.cc:359] Metadata directory not provided
I20260925 10:43:53.708072 9157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:43:53.708406 9157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:43:53.711081 9157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/master-0/data/instance:
uuid: "070d2ef9a58748dca01ca108630328c1"
format_stamp: "Formatted at 2026-09-25 10:43:53 on dist-test-slave-dxdd"
server_key: "8501b10a8cf78e5eed19e0ed6bcee7a8"
server_key_iv: "7ebd072b5a0d5de4a7748403f68eb139"
server_key_version: "encryptionkey@0"
I20260925 10:43:53.711822 9157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/master-0/wal/instance:
uuid: "070d2ef9a58748dca01ca108630328c1"
format_stamp: "Formatted at 2026-09-25 10:43:53 on dist-test-slave-dxdd"
server_key: "8501b10a8cf78e5eed19e0ed6bcee7a8"
server_key_iv: "7ebd072b5a0d5de4a7748403f68eb139"
server_key_version: "encryptionkey@0"
I20260925 10:43:53.716439 9157 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 10:43:53.719686 9171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:43:53.720919 9157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260925 10:43:53.721076 9157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/master-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/master-0/wal
uuid: "070d2ef9a58748dca01ca108630328c1"
format_stamp: "Formatted at 2026-09-25 10:43:53 on dist-test-slave-dxdd"
server_key: "8501b10a8cf78e5eed19e0ed6bcee7a8"
server_key_iv: "7ebd072b5a0d5de4a7748403f68eb139"
server_key_version: "encryptionkey@0"
I20260925 10:43:53.721271 9157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:43:53.752337 9157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:43:53.753091 9157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:43:53.753394 9157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:43:53.771081 9157 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.62:35089
I20260925 10:43:53.771100 9222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.62:35089 every 8 connection(s)
I20260925 10:43:53.772353 9157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
I20260925 10:43:53.775981 9223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:43:53.782722 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 9157
I20260925 10:43:53.782907 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/master-0/wal/instance
I20260925 10:43:53.783340 23952 external_mini_cluster.cc:1428] Setting key af2b9b20a6dda474c733cac741e4cd82
I20260925 10:43:53.786965 9223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 070d2ef9a58748dca01ca108630328c1: Bootstrap starting.
I20260925 10:43:53.790220 9223 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 070d2ef9a58748dca01ca108630328c1: Neither blocks nor log segments found. Creating new log.
I20260925 10:43:53.791486 9223 log.cc:824] T 00000000000000000000000000000000 P 070d2ef9a58748dca01ca108630328c1: Log is configured to *not* fsync() on all Append() calls
I20260925 10:43:53.794471 9223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 070d2ef9a58748dca01ca108630328c1: No bootstrap required, opened a new log
I20260925 10:43:53.796849 9226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:53.785377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39388 (local address 127.23.100.62:35089)
0925 10:43:53.786282 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0925 10:43:53.786293 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:43:53.786392 (+ 99us) server_negotiation.cc:407] Connection header received
0925 10:43:53.788107 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:43:53.788128 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:43:53.788594 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:43:53.788912 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:43:53.789495 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:53.790855 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:53.792664 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:53.793284 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:53.793411 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:53.794697 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:43:53.794710 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:43:53.794722 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:43:53.795229 (+ 507us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:43:53.795298 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:43:53.795321 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:43:53.795402 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 10:43:53.795634 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 10:43:53.795962 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":315,"threads_started":1}
I20260925 10:43:53.802604 9223 raft_consensus.cc:348] T 00000000000000000000000000000000 P 070d2ef9a58748dca01ca108630328c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070d2ef9a58748dca01ca108630328c1" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 35089 } }
I20260925 10:43:53.802930 9223 raft_consensus.cc:374] T 00000000000000000000000000000000 P 070d2ef9a58748dca01ca108630328c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:43:53.803042 9223 raft_consensus.cc:729] T 00000000000000000000000000000000 P 070d2ef9a58748dca01ca108630328c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 070d2ef9a58748dca01ca108630328c1, State: Initialized, Role: FOLLOWER
I20260925 10:43:53.803643 9223 consensus_queue.cc:260] T 00000000000000000000000000000000 P 070d2ef9a58748dca01ca108630328c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070d2ef9a58748dca01ca108630328c1" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 35089 } }
I20260925 10:43:53.803921 9223 raft_consensus.cc:388] T 00000000000000000000000000000000 P 070d2ef9a58748dca01ca108630328c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:43:53.804056 9223 raft_consensus.cc:482] T 00000000000000000000000000000000 P 070d2ef9a58748dca01ca108630328c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:43:53.804225 9223 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 070d2ef9a58748dca01ca108630328c1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:43:53.806200 9223 raft_consensus.cc:504] T 00000000000000000000000000000000 P 070d2ef9a58748dca01ca108630328c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070d2ef9a58748dca01ca108630328c1" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 35089 } }
I20260925 10:43:53.806664 9223 leader_election.cc:302] T 00000000000000000000000000000000 P 070d2ef9a58748dca01ca108630328c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 070d2ef9a58748dca01ca108630328c1; no voters:
I20260925 10:43:53.807188 9223 leader_election.cc:288] T 00000000000000000000000000000000 P 070d2ef9a58748dca01ca108630328c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:43:53.807331 9228 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 070d2ef9a58748dca01ca108630328c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:43:53.808081 9228 raft_consensus.cc:686] T 00000000000000000000000000000000 P 070d2ef9a58748dca01ca108630328c1 [term 1 LEADER]: Becoming Leader. State: Replica: 070d2ef9a58748dca01ca108630328c1, State: Running, Role: LEADER
I20260925 10:43:53.808621 9228 consensus_queue.cc:237] T 00000000000000000000000000000000 P 070d2ef9a58748dca01ca108630328c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070d2ef9a58748dca01ca108630328c1" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 35089 } }
I20260925 10:43:53.809448 9223 sys_catalog.cc:564] T 00000000000000000000000000000000 P 070d2ef9a58748dca01ca108630328c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:43:53.811406 9229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 070d2ef9a58748dca01ca108630328c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "070d2ef9a58748dca01ca108630328c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070d2ef9a58748dca01ca108630328c1" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 35089 } } }
I20260925 10:43:53.811429 9230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 070d2ef9a58748dca01ca108630328c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 070d2ef9a58748dca01ca108630328c1. Latest consensus state: current_term: 1 leader_uuid: "070d2ef9a58748dca01ca108630328c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070d2ef9a58748dca01ca108630328c1" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 35089 } } }
I20260925 10:43:53.811904 9230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 070d2ef9a58748dca01ca108630328c1 [sys.catalog]: This master's current role is: LEADER
I20260925 10:43:53.811901 9229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 070d2ef9a58748dca01ca108630328c1 [sys.catalog]: This master's current role is: LEADER
I20260925 10:43:53.813614 9234 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:43:53.818619 9234 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:43:53.827450 9234 catalog_manager.cc:1347] Generated new cluster ID: bd840aada3264396bc472b8373bc2f2a
I20260925 10:43:53.827561 9234 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:43:53.850687 9234 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:43:53.852418 9234 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:43:53.873505 9234 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 070d2ef9a58748dca01ca108630328c1: Generated new TSK 0
I20260925 10:43:53.874579 9234 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:43:53.884454 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.1:0
--local_ip_for_outbound_sockets=127.23.100.1
--webserver_interface=127.23.100.1
--webserver_port=0
--tserver_master_addrs=127.23.100.62:35089
--builtin_ntp_servers=127.23.100.20:35443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 10:43:54.233331 9247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:43:54.233626 9247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:43:54.233783 9247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:43:54.240936 9247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:43:54.241065 9247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:43:54.241245 9247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.1
I20260925 10:43:54.250727 9247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:35443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.1
--webserver_port=0
--tserver_master_addrs=127.23.100.62:35089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.1
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:43:54.251955 9247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:43:54.253656 9247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:43:54.266870 9252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:43:54.266776 9253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:43:54.266971 9255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:43:54.267693 9247 server_base.cc:1034] running on GCE node
I20260925 10:43:54.268254 9247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:43:54.269140 9247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:43:54.270393 9247 hybrid_clock.cc:648] HybridClock initialized: now 1790333034270355 us; error 48 us; skew 500 ppm
I20260925 10:43:54.270707 9247 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:43:54.272800 9247 webserver.cc:460] Webserver started at http://127.23.100.1:35481/ using document root <none> and password file <none>
I20260925 10:43:54.273465 9247 fs_manager.cc:359] Metadata directory not provided
I20260925 10:43:54.273595 9247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:43:54.273904 9247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:43:54.276671 9247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/ts-0/data/instance:
uuid: "9d29d8f7cfc648c08eb6c322c8fa2f1f"
format_stamp: "Formatted at 2026-09-25 10:43:54 on dist-test-slave-dxdd"
server_key: "230a8469fc4ce9d0b122c21782cb9787"
server_key_iv: "0d323f2ecdeaa0873e33fb3338d302c5"
server_key_version: "encryptionkey@0"
I20260925 10:43:54.277424 9247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/ts-0/wal/instance:
uuid: "9d29d8f7cfc648c08eb6c322c8fa2f1f"
format_stamp: "Formatted at 2026-09-25 10:43:54 on dist-test-slave-dxdd"
server_key: "230a8469fc4ce9d0b122c21782cb9787"
server_key_iv: "0d323f2ecdeaa0873e33fb3338d302c5"
server_key_version: "encryptionkey@0"
I20260925 10:43:54.282186 9247 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 10:43:54.285359 9261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:43:54.286695 9247 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:43:54.286859 9247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/ts-0/wal
uuid: "9d29d8f7cfc648c08eb6c322c8fa2f1f"
format_stamp: "Formatted at 2026-09-25 10:43:54 on dist-test-slave-dxdd"
server_key: "230a8469fc4ce9d0b122c21782cb9787"
server_key_iv: "0d323f2ecdeaa0873e33fb3338d302c5"
server_key_version: "encryptionkey@0"
I20260925 10:43:54.287063 9247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:43:54.313217 9247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:43:54.314034 9247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:43:54.314289 9247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:43:54.315186 9247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:43:54.316506 9247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:43:54.316588 9247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:43:54.316668 9247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:43:54.316723 9247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:43:54.347451 9247 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.1:37381
I20260925 10:43:54.347523 9373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.1:37381 every 8 connection(s)
I20260925 10:43:54.348747 9247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
I20260925 10:43:54.356611 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 9247
I20260925 10:43:54.356822 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/ts-0/wal/instance
I20260925 10:43:54.357247 23952 external_mini_cluster.cc:1428] Setting key 0920ae43d666c3fa9b08e83da8e1bdad
I20260925 10:43:54.360270 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.2:0
--local_ip_for_outbound_sockets=127.23.100.2
--webserver_interface=127.23.100.2
--webserver_port=0
--tserver_master_addrs=127.23.100.62:35089
--builtin_ntp_servers=127.23.100.20:35443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 10:43:54.361275 9376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:54.351267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:42547 (local address 127.23.100.62:35089)
0925 10:43:54.351696 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0925 10:43:54.351701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:43:54.352714 (+ 1013us) server_negotiation.cc:407] Connection header received
0925 10:43:54.354553 (+ 1839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:43:54.354558 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:43:54.354650 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:43:54.354766 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:43:54.355778 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:54.356597 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:54.358242 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:54.358893 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:54.359043 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:54.360498 (+ 1455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:43:54.360522 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:43:54.360529 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:43:54.360551 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:43:54.360606 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:43:54.360620 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:43:54.360691 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 10:43:54.360971 (+ 280us) server_negotiation.cc:299] Negotiation successful
0925 10:43:54.361043 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":228,"threads_started":1}
I20260925 10:43:54.362727 9377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:54.351736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35089 (local address 127.23.100.1:42547)
0925 10:43:54.352571 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:54.352604 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 10:43:54.354244 (+ 1640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:43:54.354989 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:43:54.354997 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:43:54.355092 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:43:54.355616 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:54.355637 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:54.356879 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:54.356886 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:54.358118 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:54.358129 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:54.359119 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:54.359125 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:54.359418 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:54.360124 (+ 706us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:43:54.360128 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:43:54.360182 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:43:54.360277 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:43:54.360735 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:43:54.360748 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:43:54.360759 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 10:43:54.360957 (+ 198us) client_negotiation.cc:240] Negotiation successful
0925 10:43:54.361357 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":261,"threads_started":1}
I20260925 10:43:54.364168 9374 heartbeater.cc:347] Connected to a master server at 127.23.100.62:35089
I20260925 10:43:54.364461 9374 heartbeater.cc:464] Registering TS with master...
I20260925 10:43:54.365170 9374 heartbeater.cc:511] Master 127.23.100.62:35089 requested a full tablet report, sending...
I20260925 10:43:54.367254 9188 ts_manager.cc:194] Registered new tserver with Master: 9d29d8f7cfc648c08eb6c322c8fa2f1f (127.23.100.1:37381)
I20260925 10:43:54.369045 9188 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.100.1:42547
W20260925 10:43:54.680819 9378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:43:54.681083 9378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:43:54.681175 9378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:43:54.688149 9378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:43:54.688244 9378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:43:54.688352 9378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.2
I20260925 10:43:54.697366 9378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:35443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.2
--webserver_port=0
--tserver_master_addrs=127.23.100.62:35089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.2
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:43:54.698637 9378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:43:54.700392 9378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:43:54.721271 9383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:43:54.721247 9384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:43:54.721480 9386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:43:54.722182 9378 server_base.cc:1034] running on GCE node
I20260925 10:43:54.722853 9378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:43:54.723795 9378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:43:54.725023 9378 hybrid_clock.cc:648] HybridClock initialized: now 1790333034724985 us; error 51 us; skew 500 ppm
I20260925 10:43:54.725358 9378 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:43:54.727648 9378 webserver.cc:460] Webserver started at http://127.23.100.2:33107/ using document root <none> and password file <none>
I20260925 10:43:54.728295 9378 fs_manager.cc:359] Metadata directory not provided
I20260925 10:43:54.728379 9378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:43:54.728645 9378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:43:54.731464 9378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/ts-1/data/instance:
uuid: "897a090c3dcd4bdeae17b6538d3e9933"
format_stamp: "Formatted at 2026-09-25 10:43:54 on dist-test-slave-dxdd"
server_key: "57f438362aee5b58a43fd10935a313b0"
server_key_iv: "84a78a32bc0895dae6ee9ecf0202112f"
server_key_version: "encryptionkey@0"
I20260925 10:43:54.732188 9378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/ts-1/wal/instance:
uuid: "897a090c3dcd4bdeae17b6538d3e9933"
format_stamp: "Formatted at 2026-09-25 10:43:54 on dist-test-slave-dxdd"
server_key: "57f438362aee5b58a43fd10935a313b0"
server_key_iv: "84a78a32bc0895dae6ee9ecf0202112f"
server_key_version: "encryptionkey@0"
I20260925 10:43:54.737112 9378 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 10:43:54.740614 9392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:43:54.741883 9378 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 10:43:54.742018 9378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/ts-1/wal
uuid: "897a090c3dcd4bdeae17b6538d3e9933"
format_stamp: "Formatted at 2026-09-25 10:43:54 on dist-test-slave-dxdd"
server_key: "57f438362aee5b58a43fd10935a313b0"
server_key_iv: "84a78a32bc0895dae6ee9ecf0202112f"
server_key_version: "encryptionkey@0"
I20260925 10:43:54.742183 9378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:43:54.768278 9378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:43:54.769105 9378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:43:54.769358 9378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:43:54.770287 9378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:43:54.772001 9378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:43:54.772086 9378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:43:54.772214 9378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:43:54.772320 9378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:43:54.806764 9378 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.2:38259
I20260925 10:43:54.806815 9504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.2:38259 every 8 connection(s)
I20260925 10:43:54.808115 9378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
I20260925 10:43:54.817382 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 9378
I20260925 10:43:54.817567 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/ts-1/wal/instance
I20260925 10:43:54.818001 23952 external_mini_cluster.cc:1428] Setting key 7dde121c00c471728e15fb231f89399a
I20260925 10:43:54.821372 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.3:0
--local_ip_for_outbound_sockets=127.23.100.3
--webserver_interface=127.23.100.3
--webserver_port=0
--tserver_master_addrs=127.23.100.62:35089
--builtin_ntp_servers=127.23.100.20:35443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 10:43:54.821533 9376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:54.810626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:35537 (local address 127.23.100.62:35089)
0925 10:43:54.810808 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 10:43:54.810812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:43:54.812315 (+ 1503us) server_negotiation.cc:407] Connection header received
0925 10:43:54.814692 (+ 2377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:43:54.814696 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:43:54.814769 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:43:54.814860 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:43:54.815916 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:54.816629 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:54.818247 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:54.818843 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:54.818984 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:54.820568 (+ 1584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:43:54.820574 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:43:54.820579 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:43:54.820602 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:43:54.820643 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:43:54.820659 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:43:54.820754 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 10:43:54.821092 (+ 338us) server_negotiation.cc:299] Negotiation successful
0925 10:43:54.821214 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 10:43:54.822568 9507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:54.811281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35089 (local address 127.23.100.2:35537)
0925 10:43:54.812161 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:54.812203 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 10:43:54.814442 (+ 2239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:43:54.815114 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:43:54.815123 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:43:54.815214 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:43:54.815767 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:54.815782 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:54.816759 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:54.816762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:54.818060 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:54.818107 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:54.819058 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:54.819063 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:54.819401 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:54.820245 (+ 844us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:43:54.820249 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:43:54.820305 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:43:54.820382 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:43:54.820853 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:43:54.820862 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:43:54.820868 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 10:43:54.821031 (+ 163us) client_negotiation.cc:240] Negotiation successful
0925 10:43:54.821329 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":259,"threads_started":1}
I20260925 10:43:54.824008 9505 heartbeater.cc:347] Connected to a master server at 127.23.100.62:35089
I20260925 10:43:54.824352 9505 heartbeater.cc:464] Registering TS with master...
I20260925 10:43:54.825253 9505 heartbeater.cc:511] Master 127.23.100.62:35089 requested a full tablet report, sending...
I20260925 10:43:54.826795 9188 ts_manager.cc:194] Registered new tserver with Master: 897a090c3dcd4bdeae17b6538d3e9933 (127.23.100.2:38259)
I20260925 10:43:54.827581 9188 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.100.2:35537
W20260925 10:43:55.140934 9508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:43:55.141192 9508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:43:55.141273 9508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:43:55.148484 9508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:43:55.148581 9508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:43:55.148684 9508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.3
I20260925 10:43:55.157438 9508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:35443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.3
--webserver_port=0
--tserver_master_addrs=127.23.100.62:35089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.3
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:43:55.158618 9508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:43:55.160310 9508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:43:55.172897 9513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:43:55.172924 9514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:43:55.172981 9516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:43:55.173619 9508 server_base.cc:1034] running on GCE node
I20260925 10:43:55.174396 9508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:43:55.175602 9508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:43:55.176826 9508 hybrid_clock.cc:648] HybridClock initialized: now 1790333035176768 us; error 51 us; skew 500 ppm
I20260925 10:43:55.177132 9508 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:43:55.179265 9508 webserver.cc:460] Webserver started at http://127.23.100.3:34459/ using document root <none> and password file <none>
I20260925 10:43:55.179929 9508 fs_manager.cc:359] Metadata directory not provided
I20260925 10:43:55.180058 9508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:43:55.180385 9508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:43:55.183600 9508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/ts-2/data/instance:
uuid: "30351654a78140afbc12f3953e058c3e"
format_stamp: "Formatted at 2026-09-25 10:43:55 on dist-test-slave-dxdd"
server_key: "da6bd0dc6aefbff87b38defc2597e703"
server_key_iv: "be64efa2071ca487fd204fbd00403287"
server_key_version: "encryptionkey@0"
I20260925 10:43:55.184546 9508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/ts-2/wal/instance:
uuid: "30351654a78140afbc12f3953e058c3e"
format_stamp: "Formatted at 2026-09-25 10:43:55 on dist-test-slave-dxdd"
server_key: "da6bd0dc6aefbff87b38defc2597e703"
server_key_iv: "be64efa2071ca487fd204fbd00403287"
server_key_version: "encryptionkey@0"
I20260925 10:43:55.189651 9508 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260925 10:43:55.193060 9522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:43:55.194239 9508 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 10:43:55.194453 9508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/ts-2/wal
uuid: "30351654a78140afbc12f3953e058c3e"
format_stamp: "Formatted at 2026-09-25 10:43:55 on dist-test-slave-dxdd"
server_key: "da6bd0dc6aefbff87b38defc2597e703"
server_key_iv: "be64efa2071ca487fd204fbd00403287"
server_key_version: "encryptionkey@0"
I20260925 10:43:55.194684 9508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:43:55.222935 9508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:43:55.223712 9508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:43:55.224053 9508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:43:55.225004 9508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:43:55.226531 9508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:43:55.226655 9508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:43:55.226792 9508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:43:55.226877 9508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:43:55.259150 9508 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.3:35591
I20260925 10:43:55.259238 9634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.3:35591 every 8 connection(s)
I20260925 10:43:55.260439 9508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
I20260925 10:43:55.267041 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 9508
I20260925 10:43:55.267246 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/ts-2/wal/instance
I20260925 10:43:55.267691 23952 external_mini_cluster.cc:1428] Setting key f041faf640c595d25112f4d60fbdcd29
I20260925 10:43:55.273016 9376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:55.262961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:60101 (local address 127.23.100.62:35089)
0925 10:43:55.263141 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 10:43:55.263145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:43:55.264488 (+ 1343us) server_negotiation.cc:407] Connection header received
0925 10:43:55.266221 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:43:55.266227 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:43:55.266364 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:43:55.266475 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:43:55.267594 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:55.268564 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:55.270415 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:55.270839 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:55.270978 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:55.272206 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:43:55.272210 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:43:55.272214 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:43:55.272230 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:43:55.272259 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:43:55.272271 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:43:55.272366 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 10:43:55.272587 (+ 221us) server_negotiation.cc:299] Negotiation successful
0925 10:43:55.272679 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 10:43:55.273970 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:55.263475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:35089 (local address 127.23.100.3:60101)
0925 10:43:55.264341 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:55.264381 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 10:43:55.265954 (+ 1573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:43:55.266705 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:43:55.266732 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:43:55.266820 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:43:55.267440 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:55.267458 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:55.268734 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:55.268741 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:55.270257 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:55.270269 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:55.271061 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:55.271069 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:55.271296 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:55.271899 (+ 603us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:43:55.271902 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:43:55.271939 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:43:55.272006 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:43:55.272365 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:43:55.272377 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:43:55.272387 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 10:43:55.272560 (+ 173us) client_negotiation.cc:240] Negotiation successful
0925 10:43:55.272938 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":262,"threads_started":1}
I20260925 10:43:55.275213 9635 heartbeater.cc:347] Connected to a master server at 127.23.100.62:35089
I20260925 10:43:55.275491 9635 heartbeater.cc:464] Registering TS with master...
I20260925 10:43:55.276141 9635 heartbeater.cc:511] Master 127.23.100.62:35089 requested a full tablet report, sending...
I20260925 10:43:55.277410 9188 ts_manager.cc:194] Registered new tserver with Master: 30351654a78140afbc12f3953e058c3e (127.23.100.3:35591)
I20260925 10:43:55.278182 9188 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.100.3:60101
I20260925 10:43:55.282269 23952 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:43:55.291978 9376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:55.286081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39404 (local address 127.23.100.62:35089)
0925 10:43:55.286286 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 10:43:55.286293 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:43:55.286706 (+ 413us) server_negotiation.cc:407] Connection header received
0925 10:43:55.286976 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:43:55.286981 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:43:55.287048 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:43:55.287152 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:43:55.287622 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:55.288285 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:55.289671 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:55.290073 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:55.290211 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:55.291332 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:43:55.291336 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:43:55.291339 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:43:55.291354 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:43:55.291383 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:43:55.291394 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:43:55.291486 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 10:43:55.291600 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 10:43:55.291703 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 10:43:55.295326 9188 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39404:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:43:55.302114 9188 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:43:55.333355 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:55.313937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.3:35591 (local address 127.0.0.1:47398)
0925 10:43:55.314995 (+ 1058us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:55.315043 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 10:43:55.315241 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:43:55.325739 (+ 10498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:43:55.325748 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:43:55.325794 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:43:55.326134 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:55.326147 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:55.328435 (+ 2288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:55.328443 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:55.330116 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:55.330132 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:55.332510 (+ 2378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:55.332514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:55.332656 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:55.332698 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 10:43:55.332832 (+ 134us) client_negotiation.cc:240] Negotiation successful
0925 10:43:55.333024 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":809,"thread_start_us":1819,"threads_started":1}
I20260925 10:43:55.333395 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:55.314488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47398 (local address 127.23.100.3:35591)
0925 10:43:55.324100 (+ 9612us) server_negotiation.cc:206] Beginning negotiation
0925 10:43:55.324111 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:43:55.324154 (+ 43us) server_negotiation.cc:407] Connection header received
0925 10:43:55.324294 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:43:55.324305 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:43:55.324621 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:43:55.326376 (+ 1755us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:43:55.326673 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:55.328299 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:55.330336 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:55.332023 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:55.332193 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:55.332416 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0925 10:43:55.332878 (+ 462us) server_negotiation.cc:299] Negotiation successful
0925 10:43:55.333038 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9407,"thread_start_us":9238,"threads_started":1}
I20260925 10:43:55.334383 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:55.314098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.2:38259 (local address 127.0.0.1:42324)
0925 10:43:55.325251 (+ 11153us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:55.325279 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 10:43:55.325397 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:43:55.327276 (+ 1879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:43:55.327281 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:43:55.327303 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:43:55.327778 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:55.327792 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:55.329660 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:55.329665 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:55.331439 (+ 1774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:55.331450 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:55.333568 (+ 2118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:55.333576 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:55.333743 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:55.333791 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 10:43:55.333869 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 10:43:55.333989 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10970,"mutex_wait_us":104,"thread_start_us":11384,"threads_started":1}
I20260925 10:43:55.334753 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:55.315290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:37381 (local address 127.0.0.1:51302)
0925 10:43:55.325445 (+ 10155us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:55.325462 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:43:55.325617 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:43:55.327511 (+ 1894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:43:55.327516 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:43:55.327541 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:43:55.327990 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:55.328002 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:55.329809 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:55.329815 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:55.332014 (+ 2199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:55.332027 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:55.333884 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:55.333891 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:55.334088 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:55.334128 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 10:43:55.334215 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 10:43:55.334375 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10054,"mutex_wait_us":38,"thread_start_us":9652,"threads_started":1}
I20260925 10:43:55.334996 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:55.315712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51302 (local address 127.23.100.1:37381)
0925 10:43:55.326102 (+ 10390us) server_negotiation.cc:206] Beginning negotiation
0925 10:43:55.326111 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:43:55.326139 (+ 28us) server_negotiation.cc:407] Connection header received
0925 10:43:55.326270 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:43:55.326280 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:43:55.327307 (+ 1027us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:43:55.327504 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:43:55.328179 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:55.329670 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:55.332193 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:55.333677 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:55.333830 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:55.334051 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0925 10:43:55.334419 (+ 368us) server_negotiation.cc:299] Negotiation successful
0925 10:43:55.334583 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10223,"thread_start_us":10102,"threads_started":1}
I20260925 10:43:55.335572 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:55.314836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42324 (local address 127.23.100.2:38259)
0925 10:43:55.326676 (+ 11840us) server_negotiation.cc:206] Beginning negotiation
0925 10:43:55.326684 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:43:55.326717 (+ 33us) server_negotiation.cc:407] Connection header received
0925 10:43:55.326818 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:43:55.326842 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:43:55.327075 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:43:55.327271 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:43:55.327950 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:55.329502 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:55.331656 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:55.333086 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:55.333223 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:55.333437 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 10:43:55.335073 (+ 1636us) server_negotiation.cc:299] Negotiation successful
0925 10:43:55.335269 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11693,"thread_start_us":11586,"threads_started":1}
I20260925 10:43:55.337862 9570 tablet_service.cc:1467] Processing CreateTablet for tablet e04e951c6fba4002bf614071bde63c08 (DEFAULT_TABLE table=test-table [id=384bf64548994181b32a518be8d38f3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:43:55.338634 9309 tablet_service.cc:1467] Processing CreateTablet for tablet e04e951c6fba4002bf614071bde63c08 (DEFAULT_TABLE table=test-table [id=384bf64548994181b32a518be8d38f3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:43:55.338721 9440 tablet_service.cc:1467] Processing CreateTablet for tablet e04e951c6fba4002bf614071bde63c08 (DEFAULT_TABLE table=test-table [id=384bf64548994181b32a518be8d38f3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:43:55.339208 9570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e04e951c6fba4002bf614071bde63c08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:43:55.339968 9440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e04e951c6fba4002bf614071bde63c08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:43:55.339965 9309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e04e951c6fba4002bf614071bde63c08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:43:55.350219 9653 tablet_bootstrap.cc:492] T e04e951c6fba4002bf614071bde63c08 P 30351654a78140afbc12f3953e058c3e: Bootstrap starting.
I20260925 10:43:55.353188 9653 tablet_bootstrap.cc:654] T e04e951c6fba4002bf614071bde63c08 P 30351654a78140afbc12f3953e058c3e: Neither blocks nor log segments found. Creating new log.
I20260925 10:43:55.354002 9654 tablet_bootstrap.cc:492] T e04e951c6fba4002bf614071bde63c08 P 897a090c3dcd4bdeae17b6538d3e9933: Bootstrap starting.
I20260925 10:43:55.355365 9653 log.cc:824] T e04e951c6fba4002bf614071bde63c08 P 30351654a78140afbc12f3953e058c3e: Log is configured to *not* fsync() on all Append() calls
I20260925 10:43:55.357577 9654 tablet_bootstrap.cc:654] T e04e951c6fba4002bf614071bde63c08 P 897a090c3dcd4bdeae17b6538d3e9933: Neither blocks nor log segments found. Creating new log.
I20260925 10:43:55.358793 9654 log.cc:824] T e04e951c6fba4002bf614071bde63c08 P 897a090c3dcd4bdeae17b6538d3e9933: Log is configured to *not* fsync() on all Append() calls
I20260925 10:43:55.361101 9654 tablet_bootstrap.cc:492] T e04e951c6fba4002bf614071bde63c08 P 897a090c3dcd4bdeae17b6538d3e9933: No bootstrap required, opened a new log
I20260925 10:43:55.361153 9653 tablet_bootstrap.cc:492] T e04e951c6fba4002bf614071bde63c08 P 30351654a78140afbc12f3953e058c3e: No bootstrap required, opened a new log
I20260925 10:43:55.361367 9654 ts_tablet_manager.cc:1397] T e04e951c6fba4002bf614071bde63c08 P 897a090c3dcd4bdeae17b6538d3e9933: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260925 10:43:55.361393 9653 ts_tablet_manager.cc:1397] T e04e951c6fba4002bf614071bde63c08 P 30351654a78140afbc12f3953e058c3e: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260925 10:43:55.366823 9657 tablet_bootstrap.cc:492] T e04e951c6fba4002bf614071bde63c08 P 9d29d8f7cfc648c08eb6c322c8fa2f1f: Bootstrap starting.
I20260925 10:43:55.370455 9657 tablet_bootstrap.cc:654] T e04e951c6fba4002bf614071bde63c08 P 9d29d8f7cfc648c08eb6c322c8fa2f1f: Neither blocks nor log segments found. Creating new log.
I20260925 10:43:55.370553 9654 raft_consensus.cc:348] T e04e951c6fba4002bf614071bde63c08 P 897a090c3dcd4bdeae17b6538d3e9933 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30351654a78140afbc12f3953e058c3e" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 35591 } } peers { permanent_uuid: "897a090c3dcd4bdeae17b6538d3e9933" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 38259 } } peers { permanent_uuid: "9d29d8f7cfc648c08eb6c322c8fa2f1f" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37381 } }
I20260925 10:43:55.371006 9654 raft_consensus.cc:374] T e04e951c6fba4002bf614071bde63c08 P 897a090c3dcd4bdeae17b6538d3e9933 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:43:55.371107 9654 raft_consensus.cc:729] T e04e951c6fba4002bf614071bde63c08 P 897a090c3dcd4bdeae17b6538d3e9933 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 897a090c3dcd4bdeae17b6538d3e9933, State: Initialized, Role: FOLLOWER
I20260925 10:43:55.370915 9653 raft_consensus.cc:348] T e04e951c6fba4002bf614071bde63c08 P 30351654a78140afbc12f3953e058c3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30351654a78140afbc12f3953e058c3e" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 35591 } } peers { permanent_uuid: "897a090c3dcd4bdeae17b6538d3e9933" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 38259 } } peers { permanent_uuid: "9d29d8f7cfc648c08eb6c322c8fa2f1f" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37381 } }
I20260925 10:43:55.371394 9653 raft_consensus.cc:374] T e04e951c6fba4002bf614071bde63c08 P 30351654a78140afbc12f3953e058c3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:43:55.371496 9653 raft_consensus.cc:729] T e04e951c6fba4002bf614071bde63c08 P 30351654a78140afbc12f3953e058c3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30351654a78140afbc12f3953e058c3e, State: Initialized, Role: FOLLOWER
I20260925 10:43:55.371716 9657 log.cc:824] T e04e951c6fba4002bf614071bde63c08 P 9d29d8f7cfc648c08eb6c322c8fa2f1f: Log is configured to *not* fsync() on all Append() calls
I20260925 10:43:55.371730 9654 consensus_queue.cc:260] T e04e951c6fba4002bf614071bde63c08 P 897a090c3dcd4bdeae17b6538d3e9933 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30351654a78140afbc12f3953e058c3e" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 35591 } } peers { permanent_uuid: "897a090c3dcd4bdeae17b6538d3e9933" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 38259 } } peers { permanent_uuid: "9d29d8f7cfc648c08eb6c322c8fa2f1f" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37381 } }
I20260925 10:43:55.372076 9653 consensus_queue.cc:260] T e04e951c6fba4002bf614071bde63c08 P 30351654a78140afbc12f3953e058c3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30351654a78140afbc12f3953e058c3e" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 35591 } } peers { permanent_uuid: "897a090c3dcd4bdeae17b6538d3e9933" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 38259 } } peers { permanent_uuid: "9d29d8f7cfc648c08eb6c322c8fa2f1f" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37381 } }
I20260925 10:43:55.372570 9374 heartbeater.cc:503] Master 127.23.100.62:35089 was elected leader, sending a full tablet report...
I20260925 10:43:55.373456 9654 ts_tablet_manager.cc:1428] T e04e951c6fba4002bf614071bde63c08 P 897a090c3dcd4bdeae17b6538d3e9933: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260925 10:43:55.373667 9505 heartbeater.cc:503] Master 127.23.100.62:35089 was elected leader, sending a full tablet report...
I20260925 10:43:55.375097 9635 heartbeater.cc:503] Master 127.23.100.62:35089 was elected leader, sending a full tablet report...
I20260925 10:43:55.376273 9653 ts_tablet_manager.cc:1428] T e04e951c6fba4002bf614071bde63c08 P 30351654a78140afbc12f3953e058c3e: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260925 10:43:55.379912 9657 tablet_bootstrap.cc:492] T e04e951c6fba4002bf614071bde63c08 P 9d29d8f7cfc648c08eb6c322c8fa2f1f: No bootstrap required, opened a new log
I20260925 10:43:55.380177 9657 ts_tablet_manager.cc:1397] T e04e951c6fba4002bf614071bde63c08 P 9d29d8f7cfc648c08eb6c322c8fa2f1f: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.006s
I20260925 10:43:55.387138 9657 raft_consensus.cc:348] T e04e951c6fba4002bf614071bde63c08 P 9d29d8f7cfc648c08eb6c322c8fa2f1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30351654a78140afbc12f3953e058c3e" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 35591 } } peers { permanent_uuid: "897a090c3dcd4bdeae17b6538d3e9933" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 38259 } } peers { permanent_uuid: "9d29d8f7cfc648c08eb6c322c8fa2f1f" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37381 } }
I20260925 10:43:55.387503 9657 raft_consensus.cc:374] T e04e951c6fba4002bf614071bde63c08 P 9d29d8f7cfc648c08eb6c322c8fa2f1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:43:55.387610 9657 raft_consensus.cc:729] T e04e951c6fba4002bf614071bde63c08 P 9d29d8f7cfc648c08eb6c322c8fa2f1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d29d8f7cfc648c08eb6c322c8fa2f1f, State: Initialized, Role: FOLLOWER
I20260925 10:43:55.388200 9657 consensus_queue.cc:260] T e04e951c6fba4002bf614071bde63c08 P 9d29d8f7cfc648c08eb6c322c8fa2f1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30351654a78140afbc12f3953e058c3e" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 35591 } } peers { permanent_uuid: "897a090c3dcd4bdeae17b6538d3e9933" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 38259 } } peers { permanent_uuid: "9d29d8f7cfc648c08eb6c322c8fa2f1f" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37381 } }
I20260925 10:43:55.389592 9657 ts_tablet_manager.cc:1428] T e04e951c6fba4002bf614071bde63c08 P 9d29d8f7cfc648c08eb6c322c8fa2f1f: Time spent starting tablet: real 0.009s user 0.009s sys 0.001s
W20260925 10:43:55.512167 9636 tablet.cc:2369] T e04e951c6fba4002bf614071bde63c08 P 30351654a78140afbc12f3953e058c3e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:43:55.560227 9506 tablet.cc:2369] T e04e951c6fba4002bf614071bde63c08 P 897a090c3dcd4bdeae17b6538d3e9933: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:43:55.601181 9375 tablet.cc:2369] T e04e951c6fba4002bf614071bde63c08 P 9d29d8f7cfc648c08eb6c322c8fa2f1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:43:55.722610 9661 raft_consensus.cc:482] T e04e951c6fba4002bf614071bde63c08 P 9d29d8f7cfc648c08eb6c322c8fa2f1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:43:55.722944 9661 raft_consensus.cc:504] T e04e951c6fba4002bf614071bde63c08 P 9d29d8f7cfc648c08eb6c322c8fa2f1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30351654a78140afbc12f3953e058c3e" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 35591 } } peers { permanent_uuid: "897a090c3dcd4bdeae17b6538d3e9933" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 38259 } } peers { permanent_uuid: "9d29d8f7cfc648c08eb6c322c8fa2f1f" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37381 } }
I20260925 10:43:55.724749 9661 leader_election.cc:288] T e04e951c6fba4002bf614071bde63c08 P 9d29d8f7cfc648c08eb6c322c8fa2f1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30351654a78140afbc12f3953e058c3e (127.23.100.3:35591), 897a090c3dcd4bdeae17b6538d3e9933 (127.23.100.2:38259)
I20260925 10:43:55.731266 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:55.725046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:36999 (local address 127.23.100.3:35591)
0925 10:43:55.725259 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 10:43:55.725268 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:43:55.725998 (+ 730us) server_negotiation.cc:407] Connection header received
0925 10:43:55.726229 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:43:55.726233 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:43:55.726290 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:43:55.726423 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:43:55.726962 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:55.727672 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:55.729648 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:55.730402 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:55.730565 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:55.730648 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 10:43:55.730911 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 10:43:55.731029 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 10:43:55.731375 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:55.724986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.3:35591 (local address 127.23.100.1:36999)
0925 10:43:55.725885 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:55.725902 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:43:55.726072 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:43:55.726439 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:43:55.726444 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:43:55.726467 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:43:55.726817 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:55.726827 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:55.727796 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:55.727804 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:55.729461 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:55.729473 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:55.730581 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:55.730587 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:55.730742 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:55.730772 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 10:43:55.730886 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 10:43:55.731011 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":779,"mutex_wait_us":1,"thread_start_us":559,"threads_started":1}
I20260925 10:43:55.731537 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:55.725196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:45651 (local address 127.23.100.2:38259)
0925 10:43:55.725427 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 10:43:55.725434 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:43:55.726233 (+ 799us) server_negotiation.cc:407] Connection header received
0925 10:43:55.726454 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:43:55.726458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:43:55.726531 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:43:55.726633 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:43:55.727203 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:55.727873 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:55.729832 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:55.730629 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:55.730806 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:55.730873 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 10:43:55.731158 (+ 285us) server_negotiation.cc:299] Negotiation successful
0925 10:43:55.731281 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 10:43:55.731608 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:55.725214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.2:38259 (local address 127.23.100.1:45651)
0925 10:43:55.726110 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:55.726125 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:43:55.726273 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:43:55.726654 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:43:55.726658 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:43:55.726681 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:43:55.727038 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:55.727046 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:55.728010 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:55.728018 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:55.729653 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:55.729665 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:55.730821 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:55.730827 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:55.730990 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:55.731012 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 10:43:55.731116 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 10:43:55.731233 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":789,"thread_start_us":553,"threads_started":1}
I20260925 10:43:55.732462 9590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e04e951c6fba4002bf614071bde63c08" candidate_uuid: "9d29d8f7cfc648c08eb6c322c8fa2f1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30351654a78140afbc12f3953e058c3e" is_pre_election: true
I20260925 10:43:55.732553 9460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e04e951c6fba4002bf614071bde63c08" candidate_uuid: "9d29d8f7cfc648c08eb6c322c8fa2f1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "897a090c3dcd4bdeae17b6538d3e9933" is_pre_election: true
I20260925 10:43:55.732975 9590 raft_consensus.cc:2454] T e04e951c6fba4002bf614071bde63c08 P 30351654a78140afbc12f3953e058c3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d29d8f7cfc648c08eb6c322c8fa2f1f in term 0.
I20260925 10:43:55.733057 9460 raft_consensus.cc:2454] T e04e951c6fba4002bf614071bde63c08 P 897a090c3dcd4bdeae17b6538d3e9933 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d29d8f7cfc648c08eb6c322c8fa2f1f in term 0.
I20260925 10:43:55.733621 9262 leader_election.cc:302] T e04e951c6fba4002bf614071bde63c08 P 9d29d8f7cfc648c08eb6c322c8fa2f1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30351654a78140afbc12f3953e058c3e, 9d29d8f7cfc648c08eb6c322c8fa2f1f; no voters:
I20260925 10:43:55.734020 9661 raft_consensus.cc:2781] T e04e951c6fba4002bf614071bde63c08 P 9d29d8f7cfc648c08eb6c322c8fa2f1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:43:55.734169 9661 raft_consensus.cc:482] T e04e951c6fba4002bf614071bde63c08 P 9d29d8f7cfc648c08eb6c322c8fa2f1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:43:55.734279 9661 raft_consensus.cc:3037] T e04e951c6fba4002bf614071bde63c08 P 9d29d8f7cfc648c08eb6c322c8fa2f1f [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:43:55.736395 9661 raft_consensus.cc:504] T e04e951c6fba4002bf614071bde63c08 P 9d29d8f7cfc648c08eb6c322c8fa2f1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30351654a78140afbc12f3953e058c3e" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 35591 } } peers { permanent_uuid: "897a090c3dcd4bdeae17b6538d3e9933" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 38259 } } peers { permanent_uuid: "9d29d8f7cfc648c08eb6c322c8fa2f1f" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37381 } }
I20260925 10:43:55.736989 9661 leader_election.cc:288] T e04e951c6fba4002bf614071bde63c08 P 9d29d8f7cfc648c08eb6c322c8fa2f1f [CANDIDATE]: Term 1 election: Requested vote from peers 30351654a78140afbc12f3953e058c3e (127.23.100.3:35591), 897a090c3dcd4bdeae17b6538d3e9933 (127.23.100.2:38259)
I20260925 10:43:55.737335 9590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e04e951c6fba4002bf614071bde63c08" candidate_uuid: "9d29d8f7cfc648c08eb6c322c8fa2f1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30351654a78140afbc12f3953e058c3e"
I20260925 10:43:55.737437 9460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e04e951c6fba4002bf614071bde63c08" candidate_uuid: "9d29d8f7cfc648c08eb6c322c8fa2f1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "897a090c3dcd4bdeae17b6538d3e9933"
I20260925 10:43:55.737558 9590 raft_consensus.cc:3037] T e04e951c6fba4002bf614071bde63c08 P 30351654a78140afbc12f3953e058c3e [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:43:55.737655 9460 raft_consensus.cc:3037] T e04e951c6fba4002bf614071bde63c08 P 897a090c3dcd4bdeae17b6538d3e9933 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:43:55.739880 9460 raft_consensus.cc:2454] T e04e951c6fba4002bf614071bde63c08 P 897a090c3dcd4bdeae17b6538d3e9933 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d29d8f7cfc648c08eb6c322c8fa2f1f in term 1.
I20260925 10:43:55.739970 9590 raft_consensus.cc:2454] T e04e951c6fba4002bf614071bde63c08 P 30351654a78140afbc12f3953e058c3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d29d8f7cfc648c08eb6c322c8fa2f1f in term 1.
I20260925 10:43:55.740298 9263 leader_election.cc:302] T e04e951c6fba4002bf614071bde63c08 P 9d29d8f7cfc648c08eb6c322c8fa2f1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 897a090c3dcd4bdeae17b6538d3e9933, 9d29d8f7cfc648c08eb6c322c8fa2f1f; no voters:
I20260925 10:43:55.740645 9661 raft_consensus.cc:2781] T e04e951c6fba4002bf614071bde63c08 P 9d29d8f7cfc648c08eb6c322c8fa2f1f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:43:55.741074 9661 raft_consensus.cc:686] T e04e951c6fba4002bf614071bde63c08 P 9d29d8f7cfc648c08eb6c322c8fa2f1f [term 1 LEADER]: Becoming Leader. State: Replica: 9d29d8f7cfc648c08eb6c322c8fa2f1f, State: Running, Role: LEADER
I20260925 10:43:55.741555 9661 consensus_queue.cc:237] T e04e951c6fba4002bf614071bde63c08 P 9d29d8f7cfc648c08eb6c322c8fa2f1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30351654a78140afbc12f3953e058c3e" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 35591 } } peers { permanent_uuid: "897a090c3dcd4bdeae17b6538d3e9933" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 38259 } } peers { permanent_uuid: "9d29d8f7cfc648c08eb6c322c8fa2f1f" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37381 } }
I20260925 10:43:55.746456 9188 catalog_manager.cc:5495] T e04e951c6fba4002bf614071bde63c08 P 9d29d8f7cfc648c08eb6c322c8fa2f1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d29d8f7cfc648c08eb6c322c8fa2f1f (127.23.100.1). New cstate: current_term: 1 leader_uuid: "9d29d8f7cfc648c08eb6c322c8fa2f1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30351654a78140afbc12f3953e058c3e" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 35591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "897a090c3dcd4bdeae17b6538d3e9933" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 38259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d29d8f7cfc648c08eb6c322c8fa2f1f" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 37381 } health_report { overall_health: HEALTHY } } }
I20260925 10:43:55.784895 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:55.776664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51308 (local address 127.23.100.1:37381)
0925 10:43:55.776907 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0925 10:43:55.776913 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:43:55.777208 (+ 295us) server_negotiation.cc:407] Connection header received
0925 10:43:55.777329 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:43:55.777334 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:43:55.777487 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:43:55.777631 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:43:55.778264 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:55.779481 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:55.781837 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:55.782608 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:55.782750 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:55.783514 (+ 764us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:43:55.784281 (+ 767us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:43:55.784449 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 10:43:55.784502 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 10:43:55.784587 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 10:43:55.794982 9460 raft_consensus.cc:1261] T e04e951c6fba4002bf614071bde63c08 P 897a090c3dcd4bdeae17b6538d3e9933 [term 1 FOLLOWER]: Refusing update from remote peer 9d29d8f7cfc648c08eb6c322c8fa2f1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:43:55.794987 9590 raft_consensus.cc:1261] T e04e951c6fba4002bf614071bde63c08 P 30351654a78140afbc12f3953e058c3e [term 1 FOLLOWER]: Refusing update from remote peer 9d29d8f7cfc648c08eb6c322c8fa2f1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:43:55.796002 9664 consensus_queue.cc:1035] T e04e951c6fba4002bf614071bde63c08 P 9d29d8f7cfc648c08eb6c322c8fa2f1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "897a090c3dcd4bdeae17b6538d3e9933" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 38259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:43:55.796408 9661 consensus_queue.cc:1035] T e04e951c6fba4002bf614071bde63c08 P 9d29d8f7cfc648c08eb6c322c8fa2f1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "30351654a78140afbc12f3953e058c3e" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:43:55.821081 9668 mvcc.cc:205] Tried to move back new op lower bound from 7333204114604380160 to 7333204114401222656. Current Snapshot: MvccSnapshot[applied={T|T < 7333204114604380160}]
I20260925 10:43:55.845821 9186 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:43:55.846153 9186 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:43:55.849509 9186 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 070d2ef9a58748dca01ca108630328c1: Sending DeleteTablet for 3 replicas of tablet e04e951c6fba4002bf614071bde63c08
I20260925 10:43:55.850628 9570 tablet_service.cc:1514] Processing DeleteTablet for tablet e04e951c6fba4002bf614071bde63c08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:43:55 UTC) from {username='slave'} at 127.0.0.1:47398
I20260925 10:43:55.850662 9309 tablet_service.cc:1514] Processing DeleteTablet for tablet e04e951c6fba4002bf614071bde63c08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:43:55 UTC) from {username='slave'} at 127.0.0.1:51302
I20260925 10:43:55.850667 9440 tablet_service.cc:1514] Processing DeleteTablet for tablet e04e951c6fba4002bf614071bde63c08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:43:55 UTC) from {username='slave'} at 127.0.0.1:42324
I20260925 10:43:55.852619 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 9247
I20260925 10:43:55.861681 9678 tablet_replica.cc:330] T e04e951c6fba4002bf614071bde63c08 P 9d29d8f7cfc648c08eb6c322c8fa2f1f: stopping tablet replica
I20260925 10:43:55.861753 9676 tablet_replica.cc:330] T e04e951c6fba4002bf614071bde63c08 P 30351654a78140afbc12f3953e058c3e: stopping tablet replica
I20260925 10:43:55.862106 9678 raft_consensus.cc:2229] T e04e951c6fba4002bf614071bde63c08 P 9d29d8f7cfc648c08eb6c322c8fa2f1f [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:43:55.862098 9676 raft_consensus.cc:2229] T e04e951c6fba4002bf614071bde63c08 P 30351654a78140afbc12f3953e058c3e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:43:55.862622 9676 raft_consensus.cc:2258] T e04e951c6fba4002bf614071bde63c08 P 30351654a78140afbc12f3953e058c3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:43:55.862864 9678 raft_consensus.cc:2258] T e04e951c6fba4002bf614071bde63c08 P 9d29d8f7cfc648c08eb6c322c8fa2f1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:43:55.865483 9676 ts_tablet_manager.cc:1905] T e04e951c6fba4002bf614071bde63c08 P 30351654a78140afbc12f3953e058c3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:43:55.868211 9678 ts_tablet_manager.cc:1905] T e04e951c6fba4002bf614071bde63c08 P 9d29d8f7cfc648c08eb6c322c8fa2f1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:43:55.870070 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:55.853910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51316 (local address 127.23.100.1:37381)
0925 10:43:55.854077 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 10:43:55.854082 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:43:55.859951 (+ 5869us) server_negotiation.cc:407] Connection header received
0925 10:43:55.860945 (+ 994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:43:55.860951 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:43:55.861066 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:43:55.861171 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:43:55.862239 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:55.863525 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:55.865992 (+ 2467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:55.866686 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:55.866783 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:55.868187 (+ 1404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:43:55.868213 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:43:55.868228 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:43:55.869001 (+ 773us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:43:55.869060 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:43:55.869090 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:43:55.869172 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 10:43:55.869474 (+ 302us) server_negotiation.cc:299] Negotiation successful
0925 10:43:55.869573 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 10:43:55.871753 9677 tablet_replica.cc:330] T e04e951c6fba4002bf614071bde63c08 P 897a090c3dcd4bdeae17b6538d3e9933: stopping tablet replica
I20260925 10:43:55.871909 9369 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:43:55.872264 9677 raft_consensus.cc:2229] T e04e951c6fba4002bf614071bde63c08 P 897a090c3dcd4bdeae17b6538d3e9933 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:43:55.872854 9677 raft_consensus.cc:2258] T e04e951c6fba4002bf614071bde63c08 P 897a090c3dcd4bdeae17b6538d3e9933 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:43:55.873138 9676 ts_tablet_manager.cc:1918] T e04e951c6fba4002bf614071bde63c08 P 30351654a78140afbc12f3953e058c3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:43:55.873793 9676 log.cc:1196] T e04e951c6fba4002bf614071bde63c08 P 30351654a78140afbc12f3953e058c3e: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/ts-2/wal/wals/e04e951c6fba4002bf614071bde63c08
I20260925 10:43:55.874413 9676 ts_tablet_manager.cc:1939] T e04e951c6fba4002bf614071bde63c08 P 30351654a78140afbc12f3953e058c3e: Deleting consensus metadata
I20260925 10:43:55.875056 9677 ts_tablet_manager.cc:1905] T e04e951c6fba4002bf614071bde63c08 P 897a090c3dcd4bdeae17b6538d3e9933: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:43:55.876307 9172 catalog_manager.cc:4841] TS 30351654a78140afbc12f3953e058c3e (127.23.100.3:35591): tablet e04e951c6fba4002bf614071bde63c08 (table test-table [id=384bf64548994181b32a518be8d38f3a]) successfully deleted
I20260925 10:43:55.881814 9677 ts_tablet_manager.cc:1918] T e04e951c6fba4002bf614071bde63c08 P 897a090c3dcd4bdeae17b6538d3e9933: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:43:55.881978 9677 log.cc:1196] T e04e951c6fba4002bf614071bde63c08 P 897a090c3dcd4bdeae17b6538d3e9933: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790332740062128-23952-0/minicluster-data/ts-1/wal/wals/e04e951c6fba4002bf614071bde63c08
I20260925 10:43:55.882537 9677 ts_tablet_manager.cc:1939] T e04e951c6fba4002bf614071bde63c08 P 897a090c3dcd4bdeae17b6538d3e9933: Deleting consensus metadata
I20260925 10:43:55.883945 9173 catalog_manager.cc:4841] TS 897a090c3dcd4bdeae17b6538d3e9933 (127.23.100.2:38259): tablet e04e951c6fba4002bf614071bde63c08 (table test-table [id=384bf64548994181b32a518be8d38f3a]) successfully deleted
I20260925 10:43:55.973949 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 9247
I20260925 10:43:55.992960 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 9378
W20260925 10:43:55.993042 9174 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.100.1:37381 (error 108)
I20260925 10:43:55.995254 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:55.994427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:37381 (local address 127.0.0.1:51326)
0925 10:43:55.994922 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:55.995073 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.1:37381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":254,"threads_started":1}
W20260925 10:43:55.995483 9174 catalog_manager.cc:4584] TS 9d29d8f7cfc648c08eb6c322c8fa2f1f (127.23.100.1:37381): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e04e951c6fba4002bf614071bde63c08: Network error: Client connection negotiation failed: client connection to 127.23.100.1:37381: connect: Connection refused (error 111)
I20260925 10:43:56.000738 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:55.993856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42332 (local address 127.23.100.2:38259)
0925 10:43:55.994067 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 10:43:55.994074 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:43:55.994192 (+ 118us) server_negotiation.cc:407] Connection header received
0925 10:43:55.994422 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:43:55.994426 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:43:55.994530 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:43:55.994663 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:43:55.995120 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:55.996026 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:55.997339 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:55.997764 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:55.997900 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:55.999327 (+ 1427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:43:55.999336 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:43:55.999350 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:43:55.999951 (+ 601us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:43:56.000004 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:43:56.000026 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:43:56.000155 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 10:43:56.000293 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 10:43:56.000405 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 10:43:56.001398 9500 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:43:56.049151 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:56.048601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:37381 (local address 127.0.0.1:51332)
0925 10:43:56.048851 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:56.048973 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.1:37381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260925 10:43:56.050249 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:56.049814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:37381 (local address 127.0.0.1:51340)
0925 10:43:56.049979 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:56.050073 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.1:37381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 10:43:56.090855 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 9378
I20260925 10:43:56.108129 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 9508
I20260925 10:43:56.110519 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:56.110083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:37381 (local address 127.0.0.1:51350)
0925 10:43:56.110233 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:56.110371 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.1:37381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 10:43:56.111519 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:56.111201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:37381 (local address 127.0.0.1:51362)
0925 10:43:56.111327 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:56.111391 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.1:37381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 10:43:56.115391 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:56.108927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47412 (local address 127.23.100.3:35591)
0925 10:43:56.109093 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 10:43:56.109097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:43:56.109115 (+ 18us) server_negotiation.cc:407] Connection header received
0925 10:43:56.109320 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:43:56.109324 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:43:56.109406 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:43:56.109499 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:43:56.109884 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:56.110777 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:56.112313 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:56.112733 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:56.112868 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:56.113971 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:43:56.113980 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:43:56.113995 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:43:56.114659 (+ 664us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:43:56.114706 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:43:56.114725 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:43:56.114854 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 10:43:56.114982 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 10:43:56.115088 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 10:43:56.116063 9630 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:43:56.192345 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:56.191756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:37381 (local address 127.0.0.1:51370)
0925 10:43:56.192023 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:56.192145 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.1:37381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 10:43:56.195612 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:56.193040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:37381 (local address 127.0.0.1:51372)
0925 10:43:56.195345 (+ 2305us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:56.195435 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.1:37381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2209,"threads_started":1}
I20260925 10:43:56.209079 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 9508
I20260925 10:43:56.225934 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 9157
I20260925 10:43:56.227077 9218 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:43:56.333396 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 9157
2026-09-25T10:43:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3054 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:43:56 dist-test-slave-dxdd krb5kdc[9689](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:43:56 dist-test-slave-dxdd krb5kdc[9689](info): set up 2 sockets
Sep 25 10:43:56 dist-test-slave-dxdd krb5kdc[9689](info): commencing operation
krb5kdc: starting...
W20260925 10:43:57.422519 23952 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Sep 25 10:43:57 dist-test-slave-dxdd krb5kdc[9689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:43:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:43:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
I20260925 10:43:57.657074 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:38169
--webserver_interface=127.23.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:36413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.62:38169
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:43:57.980384 9702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:43:57.980659 9702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:43:57.980746 9702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:43:57.987829 9702 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:43:57.987928 9702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:43:57.987982 9702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:43:57.988037 9702 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:43:57.988080 9702 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:43:57.997654 9702 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:36413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.62:38169
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:38169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:43:57.998989 9702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:43:58.000229 9702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:43:58.011824 9707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:43:58.011768 9708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:43:58.011742 9710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:43:58.012522 9702 server_base.cc:1034] running on GCE node
I20260925 10:43:58.013207 9702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:43:58.014540 9702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:43:58.015766 9702 hybrid_clock.cc:648] HybridClock initialized: now 1790333038015731 us; error 47 us; skew 500 ppm
I20260925 10:43:58.016106 9702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:43:58 dist-test-slave-dxdd krb5kdc[9689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333038, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:43:58.019274 9702 init.cc:377] Logged in from keytab as kudu/127.23.100.62@KRBTEST.COM (short username kudu)
I20260925 10:43:58.020707 9702 webserver.cc:460] Webserver started at http://127.23.100.62:36151/ using document root <none> and password file <none>
I20260925 10:43:58.021355 9702 fs_manager.cc:359] Metadata directory not provided
I20260925 10:43:58.021456 9702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:43:58.021699 9702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:43:58.024636 9702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/master-0/data/instance:
uuid: "7fc3832077344c988678da316c5250e9"
format_stamp: "Formatted at 2026-09-25 10:43:58 on dist-test-slave-dxdd"
server_key: "2dc915a75388865b6bb434b426620278"
server_key_iv: "4be85d5016e8a8d2b3257a88bd15ce62"
server_key_version: "encryptionkey@0"
I20260925 10:43:58.025339 9702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/master-0/wal/instance:
uuid: "7fc3832077344c988678da316c5250e9"
format_stamp: "Formatted at 2026-09-25 10:43:58 on dist-test-slave-dxdd"
server_key: "2dc915a75388865b6bb434b426620278"
server_key_iv: "4be85d5016e8a8d2b3257a88bd15ce62"
server_key_version: "encryptionkey@0"
I20260925 10:43:58.032624 9702 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260925 10:43:58.036531 9717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:43:58.037817 9702 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:43:58.037957 9702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/master-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/master-0/wal
uuid: "7fc3832077344c988678da316c5250e9"
format_stamp: "Formatted at 2026-09-25 10:43:58 on dist-test-slave-dxdd"
server_key: "2dc915a75388865b6bb434b426620278"
server_key_iv: "4be85d5016e8a8d2b3257a88bd15ce62"
server_key_version: "encryptionkey@0"
I20260925 10:43:58.038115 9702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:43:58.053051 9702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:43:58.057354 9702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:43:58.057575 9702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:43:58.075608 9702 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.62:38169
I20260925 10:43:58.075650 9768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.62:38169 every 8 connection(s)
I20260925 10:43:58.076977 9702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
I20260925 10:43:58.079803 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 9702
I20260925 10:43:58.079984 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/master-0/wal/instance
I20260925 10:43:58.080447 23952 external_mini_cluster.cc:1428] Setting key 07e33f8d79a2ac71419e1e9e0c482852
I20260925 10:43:58.080825 9769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:43:58.094982 9769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fc3832077344c988678da316c5250e9: Bootstrap starting.
Sep 25 10:43:58 dist-test-slave-dxdd krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:43:58.098357 9769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7fc3832077344c988678da316c5250e9: Neither blocks nor log segments found. Creating new log.
I20260925 10:43:58.099468 9769 log.cc:824] T 00000000000000000000000000000000 P 7fc3832077344c988678da316c5250e9: Log is configured to *not* fsync() on all Append() calls
I20260925 10:43:58.102147 9769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fc3832077344c988678da316c5250e9: No bootstrap required, opened a new log
I20260925 10:43:58.103755 9772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:58.082162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55948 (local address 127.23.100.62:38169)
0925 10:43:58.082982 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0925 10:43:58.082994 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:43:58.083031 (+ 37us) server_negotiation.cc:407] Connection header received
0925 10:43:58.085238 (+ 2207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:43:58.085258 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:43:58.085633 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:43:58.085938 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:43:58.087746 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:58.089002 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:58.090906 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:58.091610 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:58.091742 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:58.096656 (+ 4914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:43:58.096673 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:43:58.096687 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:43:58.096727 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:43:58.099865 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:43:58.100576 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:43:58.100583 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:43:58.100588 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:43:58.100760 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:43:58.101151 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:43:58.101161 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:43:58.101164 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:43:58.101643 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:43:58.101977 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0925 10:43:58.102417 (+ 440us) server_negotiation.cc:299] Negotiation successful
0925 10:43:58.102836 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":423,"threads_started":1}
I20260925 10:43:58.109982 9769 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7fc3832077344c988678da316c5250e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fc3832077344c988678da316c5250e9" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 38169 } }
I20260925 10:43:58.110379 9769 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7fc3832077344c988678da316c5250e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:43:58.110496 9769 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7fc3832077344c988678da316c5250e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fc3832077344c988678da316c5250e9, State: Initialized, Role: FOLLOWER
I20260925 10:43:58.111146 9769 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7fc3832077344c988678da316c5250e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fc3832077344c988678da316c5250e9" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 38169 } }
I20260925 10:43:58.111392 9769 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7fc3832077344c988678da316c5250e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:43:58.111553 9769 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7fc3832077344c988678da316c5250e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:43:58.111747 9769 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7fc3832077344c988678da316c5250e9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:43:58.113823 9769 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7fc3832077344c988678da316c5250e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fc3832077344c988678da316c5250e9" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 38169 } }
I20260925 10:43:58.114293 9769 leader_election.cc:302] T 00000000000000000000000000000000 P 7fc3832077344c988678da316c5250e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7fc3832077344c988678da316c5250e9; no voters:
I20260925 10:43:58.114905 9769 leader_election.cc:288] T 00000000000000000000000000000000 P 7fc3832077344c988678da316c5250e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:43:58.115032 9774 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7fc3832077344c988678da316c5250e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:43:58.115710 9774 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7fc3832077344c988678da316c5250e9 [term 1 LEADER]: Becoming Leader. State: Replica: 7fc3832077344c988678da316c5250e9, State: Running, Role: LEADER
I20260925 10:43:58.116271 9774 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7fc3832077344c988678da316c5250e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fc3832077344c988678da316c5250e9" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 38169 } }
I20260925 10:43:58.116894 9769 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7fc3832077344c988678da316c5250e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:43:58.119096 9775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fc3832077344c988678da316c5250e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7fc3832077344c988678da316c5250e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fc3832077344c988678da316c5250e9" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 38169 } } }
I20260925 10:43:58.119805 9775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fc3832077344c988678da316c5250e9 [sys.catalog]: This master's current role is: LEADER
I20260925 10:43:58.119203 9776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fc3832077344c988678da316c5250e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7fc3832077344c988678da316c5250e9. Latest consensus state: current_term: 1 leader_uuid: "7fc3832077344c988678da316c5250e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fc3832077344c988678da316c5250e9" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 38169 } } }
I20260925 10:43:58.120288 9776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fc3832077344c988678da316c5250e9 [sys.catalog]: This master's current role is: LEADER
I20260925 10:43:58.122251 9783 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:43:58.127285 9783 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:43:58.135049 9783 catalog_manager.cc:1347] Generated new cluster ID: 076d885b109c4533a5e09a3716e24dac
I20260925 10:43:58.135159 9783 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:43:58.151638 9783 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:43:58.152896 9783 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:43:58.159394 9783 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7fc3832077344c988678da316c5250e9: Generated new TSK 0
I20260925 10:43:58.160207 9783 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab.
Entry for principal kudu/127.23.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab.
I20260925 10:43:58.238000 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.1:0
--local_ip_for_outbound_sockets=127.23.100.1
--webserver_interface=127.23.100.1
--webserver_port=0
--tserver_master_addrs=127.23.100.62:38169
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab
--principal=kudu/127.23.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:36413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:43:58.571017 9795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:43:58.571287 9795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:43:58.571467 9795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:43:58.578548 9795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:43:58.578661 9795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:43:58.578835 9795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.1
I20260925 10:43:58.588074 9795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:36413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab
--principal=kudu/127.23.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.1
--webserver_port=0
--tserver_master_addrs=127.23.100.62:38169
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.1
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:43:58.589211 9795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:43:58.590498 9795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:43:58.603479 9803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:43:58.603542 9800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:43:58.603576 9801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:43:58.604359 9795 server_base.cc:1034] running on GCE node
I20260925 10:43:58.604910 9795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:43:58.605795 9795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:43:58.607014 9795 hybrid_clock.cc:648] HybridClock initialized: now 1790333038606970 us; error 54 us; skew 500 ppm
I20260925 10:43:58.607318 9795 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:43:58 dist-test-slave-dxdd krb5kdc[9689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333038, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:43:58.610436 9795 init.cc:377] Logged in from keytab as kudu/127.23.100.1@KRBTEST.COM (short username kudu)
I20260925 10:43:58.611779 9795 webserver.cc:460] Webserver started at http://127.23.100.1:44369/ using document root <none> and password file <none>
I20260925 10:43:58.612475 9795 fs_manager.cc:359] Metadata directory not provided
I20260925 10:43:58.612632 9795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:43:58.612943 9795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:43:58.616031 9795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/ts-0/data/instance:
uuid: "7938af616b92449ab7ce783ce3cf1857"
format_stamp: "Formatted at 2026-09-25 10:43:58 on dist-test-slave-dxdd"
server_key: "77ec2379759a78d83259b421ee7e6e32"
server_key_iv: "bc4381a6fe53806f66df9f981057605b"
server_key_version: "encryptionkey@0"
I20260925 10:43:58.616819 9795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/ts-0/wal/instance:
uuid: "7938af616b92449ab7ce783ce3cf1857"
format_stamp: "Formatted at 2026-09-25 10:43:58 on dist-test-slave-dxdd"
server_key: "77ec2379759a78d83259b421ee7e6e32"
server_key_iv: "bc4381a6fe53806f66df9f981057605b"
server_key_version: "encryptionkey@0"
I20260925 10:43:58.622687 9795 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 10:43:58.626194 9810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:43:58.627453 9795 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 10:43:58.627637 9795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/ts-0/wal
uuid: "7938af616b92449ab7ce783ce3cf1857"
format_stamp: "Formatted at 2026-09-25 10:43:58 on dist-test-slave-dxdd"
server_key: "77ec2379759a78d83259b421ee7e6e32"
server_key_iv: "bc4381a6fe53806f66df9f981057605b"
server_key_version: "encryptionkey@0"
I20260925 10:43:58.627839 9795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:43:58.643147 9795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:43:58.647912 9795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:43:58.648157 9795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:43:58.649009 9795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:43:58.650240 9795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:43:58.650370 9795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:43:58.650484 9795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:43:58.650532 9795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:43:58.683367 9795 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.1:33183
I20260925 10:43:58.683409 9922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.1:33183 every 8 connection(s)
I20260925 10:43:58.684712 9795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
I20260925 10:43:58.685557 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 9795
I20260925 10:43:58.685766 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/ts-0/wal/instance
I20260925 10:43:58.686328 23952 external_mini_cluster.cc:1428] Setting key 5dc609535fb052f218739e0bc4544418
Sep 25 10:43:58 dist-test-slave-dxdd krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333038, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.1@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:43:58.707046 9925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:58.687639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:38673 (local address 127.23.100.62:38169)
0925 10:43:58.688114 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0925 10:43:58.688118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:43:58.689400 (+ 1282us) server_negotiation.cc:407] Connection header received
0925 10:43:58.692320 (+ 2920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:43:58.692324 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:43:58.692387 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:43:58.692496 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:43:58.694535 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:58.695240 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:58.697638 (+ 2398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:58.698040 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:58.698139 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:58.701577 (+ 3438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:43:58.701582 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:43:58.701586 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:43:58.701615 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:43:58.704125 (+ 2510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:43:58.704803 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:43:58.704814 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:43:58.704818 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:43:58.704929 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:43:58.705392 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:43:58.705397 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:43:58.705400 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:43:58.705649 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:43:58.705810 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 10:43:58.706455 (+ 645us) server_negotiation.cc:299] Negotiation successful
0925 10:43:58.706668 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":253,"threads_started":1}
I20260925 10:43:58.707945 9926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:58.688314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:38169 (local address 127.23.100.1:38673)
0925 10:43:58.689221 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:58.689272 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 10:43:58.692029 (+ 2757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:43:58.692744 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:43:58.692757 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:43:58.693543 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:43:58.694355 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:58.694383 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:58.695416 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:58.695424 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:58.697483 (+ 2059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:58.697499 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:58.698262 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:58.698270 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:58.698532 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:58.699256 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:43:58.699276 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:43:58.701339 (+ 2063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:43:58.704259 (+ 2920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:43:58.704267 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:43:58.704281 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:43:58.704680 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:43:58.705086 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:43:58.705091 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:43:58.705094 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:43:58.705274 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:43:58.705775 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:43:58.705782 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:43:58.706090 (+ 308us) client_negotiation.cc:769] Sending connection context
0925 10:43:58.706351 (+ 261us) client_negotiation.cc:240] Negotiation successful
0925 10:43:58.706830 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":324,"threads_started":1}
I20260925 10:43:58.709558 9923 heartbeater.cc:347] Connected to a master server at 127.23.100.62:38169
I20260925 10:43:58.709893 9923 heartbeater.cc:464] Registering TS with master...
I20260925 10:43:58.710853 9923 heartbeater.cc:511] Master 127.23.100.62:38169 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.100.2@KRBTEST.COM; defaulting to no policy
I20260925 10:43:58.713130 9734 ts_manager.cc:194] Registered new tserver with Master: 7938af616b92449ab7ce783ce3cf1857 (127.23.100.1:33183)
I20260925 10:43:58.714689 9734 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.1@KRBTEST.COM'} at 127.23.100.1:38673
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab.
Entry for principal kudu/127.23.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab.
I20260925 10:43:58.742797 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.2:0
--local_ip_for_outbound_sockets=127.23.100.2
--webserver_interface=127.23.100.2
--webserver_port=0
--tserver_master_addrs=127.23.100.62:38169
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab
--principal=kudu/127.23.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:36413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:43:59.061165 9929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:43:59.061419 9929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:43:59.061602 9929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:43:59.068732 9929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:43:59.068830 9929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:43:59.069034 9929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.2
I20260925 10:43:59.078634 9929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:36413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab
--principal=kudu/127.23.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.2
--webserver_port=0
--tserver_master_addrs=127.23.100.62:38169
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.2
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:43:59.079851 9929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:43:59.081183 9929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:43:59.093868 9934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:43:59.093907 9935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:43:59.093824 9937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:43:59.094594 9929 server_base.cc:1034] running on GCE node
I20260925 10:43:59.095124 9929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:43:59.095945 9929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:43:59.097183 9929 hybrid_clock.cc:648] HybridClock initialized: now 1790333039097132 us; error 41 us; skew 500 ppm
I20260925 10:43:59.097520 9929 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:43:59 dist-test-slave-dxdd krb5kdc[9689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333039, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:43:59.100509 9929 init.cc:377] Logged in from keytab as kudu/127.23.100.2@KRBTEST.COM (short username kudu)
I20260925 10:43:59.101904 9929 webserver.cc:460] Webserver started at http://127.23.100.2:33991/ using document root <none> and password file <none>
I20260925 10:43:59.102617 9929 fs_manager.cc:359] Metadata directory not provided
I20260925 10:43:59.102761 9929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:43:59.103083 9929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:43:59.105786 9929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/ts-1/data/instance:
uuid: "9f82eaf4bcef48cbba3320263e7dbee9"
format_stamp: "Formatted at 2026-09-25 10:43:59 on dist-test-slave-dxdd"
server_key: "4eaef8b3fc3e6387cdc18ab52a8c0276"
server_key_iv: "2e95cc717d28f8a95c9b1f320ef5098f"
server_key_version: "encryptionkey@0"
I20260925 10:43:59.106606 9929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/ts-1/wal/instance:
uuid: "9f82eaf4bcef48cbba3320263e7dbee9"
format_stamp: "Formatted at 2026-09-25 10:43:59 on dist-test-slave-dxdd"
server_key: "4eaef8b3fc3e6387cdc18ab52a8c0276"
server_key_iv: "2e95cc717d28f8a95c9b1f320ef5098f"
server_key_version: "encryptionkey@0"
I20260925 10:43:59.111523 9929 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 10:43:59.114866 9944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:43:59.115993 9929 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 10:43:59.116178 9929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/ts-1/wal
uuid: "9f82eaf4bcef48cbba3320263e7dbee9"
format_stamp: "Formatted at 2026-09-25 10:43:59 on dist-test-slave-dxdd"
server_key: "4eaef8b3fc3e6387cdc18ab52a8c0276"
server_key_iv: "2e95cc717d28f8a95c9b1f320ef5098f"
server_key_version: "encryptionkey@0"
I20260925 10:43:59.116364 9929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:43:59.135447 9929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:43:59.139068 9929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:43:59.139369 9929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:43:59.140267 9929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:43:59.141491 9929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:43:59.141590 9929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:43:59.141746 9929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:43:59.141819 9929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:43:59.174713 9929 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.2:40277
I20260925 10:43:59.174794 10056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.2:40277 every 8 connection(s)
I20260925 10:43:59.176016 9929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
I20260925 10:43:59.184371 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 9929
I20260925 10:43:59.184561 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/ts-1/wal/instance
I20260925 10:43:59.185101 23952 external_mini_cluster.cc:1428] Setting key 6484d299d61449ade7eba09f00a6285c
Sep 25 10:43:59 dist-test-slave-dxdd krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333039, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.2@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:43:59.195998 9925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:59.178559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:46343 (local address 127.23.100.62:38169)
0925 10:43:59.178740 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 10:43:59.178745 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:43:59.179894 (+ 1149us) server_negotiation.cc:407] Connection header received
0925 10:43:59.181620 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:43:59.181624 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:43:59.181693 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:43:59.181783 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:43:59.183347 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:59.184026 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:59.186071 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:59.186600 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:59.186706 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:59.190572 (+ 3866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:43:59.190577 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:43:59.190580 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:43:59.190611 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:43:59.193135 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:43:59.193857 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:43:59.193874 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:43:59.193876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:43:59.193996 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:43:59.194433 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:43:59.194437 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:43:59.194439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:43:59.194675 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:43:59.194829 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 10:43:59.195453 (+ 624us) server_negotiation.cc:299] Negotiation successful
0925 10:43:59.195662 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 10:43:59.196913 10059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:59.179074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:38169 (local address 127.23.100.2:46343)
0925 10:43:59.179735 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:59.179777 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 10:43:59.181377 (+ 1600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:43:59.182016 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:43:59.182032 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:43:59.182644 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:43:59.183205 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:59.183221 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:59.184176 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:59.184182 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:59.185948 (+ 1766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:59.185960 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:59.186824 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:59.186832 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:59.187060 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:59.187718 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:43:59.187751 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:43:59.190368 (+ 2617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:43:59.193286 (+ 2918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:43:59.193295 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:43:59.193309 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:43:59.193724 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:43:59.194103 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:43:59.194109 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:43:59.194112 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:43:59.194287 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:43:59.194796 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:43:59.194803 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:43:59.195116 (+ 313us) client_negotiation.cc:769] Sending connection context
0925 10:43:59.195358 (+ 242us) client_negotiation.cc:240] Negotiation successful
0925 10:43:59.195849 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":254,"threads_started":1}
I20260925 10:43:59.198359 10057 heartbeater.cc:347] Connected to a master server at 127.23.100.62:38169
I20260925 10:43:59.198691 10057 heartbeater.cc:464] Registering TS with master...
I20260925 10:43:59.199581 10057 heartbeater.cc:511] Master 127.23.100.62:38169 requested a full tablet report, sending...
I20260925 10:43:59.201216 9734 ts_manager.cc:194] Registered new tserver with Master: 9f82eaf4bcef48cbba3320263e7dbee9 (127.23.100.2:40277)
I20260925 10:43:59.202471 9734 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.2@KRBTEST.COM'} at 127.23.100.2:46343
WARNING: no policy specified for kudu/127.23.100.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab.
Entry for principal kudu/127.23.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab.
I20260925 10:43:59.242138 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.3:0
--local_ip_for_outbound_sockets=127.23.100.3
--webserver_interface=127.23.100.3
--webserver_port=0
--tserver_master_addrs=127.23.100.62:38169
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab
--principal=kudu/127.23.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:36413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:43:59.559221 10062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:43:59.559470 10062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:43:59.559561 10062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:43:59.566648 10062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:43:59.566744 10062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:43:59.566851 10062 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.3
I20260925 10:43:59.576092 10062 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:36413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab
--principal=kudu/127.23.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.3
--webserver_port=0
--tserver_master_addrs=127.23.100.62:38169
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.3
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:43:59.577155 10062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:43:59.578330 10062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:43:59.590766 10068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:43:59.590806 10070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:43:59.590843 10067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:43:59.591965 10062 server_base.cc:1034] running on GCE node
I20260925 10:43:59.592588 10062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:43:59.593555 10062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:43:59.594749 10062 hybrid_clock.cc:648] HybridClock initialized: now 1790333039594719 us; error 25 us; skew 500 ppm
I20260925 10:43:59.595072 10062 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:43:59 dist-test-slave-dxdd krb5kdc[9689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333039, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:43:59.597966 10062 init.cc:377] Logged in from keytab as kudu/127.23.100.3@KRBTEST.COM (short username kudu)
I20260925 10:43:59.599475 10062 webserver.cc:460] Webserver started at http://127.23.100.3:33643/ using document root <none> and password file <none>
I20260925 10:43:59.600136 10062 fs_manager.cc:359] Metadata directory not provided
I20260925 10:43:59.600289 10062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:43:59.600585 10062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:43:59.603534 10062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/ts-2/data/instance:
uuid: "208cd03307d64d96b9e5dcea1bf93f74"
format_stamp: "Formatted at 2026-09-25 10:43:59 on dist-test-slave-dxdd"
server_key: "6b0b96946bc4d0c5ef188f3b349c2fb4"
server_key_iv: "7b964bac9862938d1a5162196aca0e34"
server_key_version: "encryptionkey@0"
I20260925 10:43:59.604298 10062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/ts-2/wal/instance:
uuid: "208cd03307d64d96b9e5dcea1bf93f74"
format_stamp: "Formatted at 2026-09-25 10:43:59 on dist-test-slave-dxdd"
server_key: "6b0b96946bc4d0c5ef188f3b349c2fb4"
server_key_iv: "7b964bac9862938d1a5162196aca0e34"
server_key_version: "encryptionkey@0"
I20260925 10:43:59.609315 10062 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 10:43:59.612607 10077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:43:59.613941 10062 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 10:43:59.614113 10062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/ts-2/wal
uuid: "208cd03307d64d96b9e5dcea1bf93f74"
format_stamp: "Formatted at 2026-09-25 10:43:59 on dist-test-slave-dxdd"
server_key: "6b0b96946bc4d0c5ef188f3b349c2fb4"
server_key_iv: "7b964bac9862938d1a5162196aca0e34"
server_key_version: "encryptionkey@0"
I20260925 10:43:59.614305 10062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:43:59.635839 10062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:43:59.639619 10062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:43:59.639892 10062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:43:59.640782 10062 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:43:59.642104 10062 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:43:59.642207 10062 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:43:59.642381 10062 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:43:59.642457 10062 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260925 10:43:59.675418 10062 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.3:38783
I20260925 10:43:59.675473 10189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.3:38783 every 8 connection(s)
I20260925 10:43:59.676687 10062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
I20260925 10:43:59.684587 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 10062
I20260925 10:43:59.684770 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/ts-2/wal/instance
I20260925 10:43:59.685285 23952 external_mini_cluster.cc:1428] Setting key 4121bcbe41eefaefc532a5111eb6059e
Sep 25 10:43:59 dist-test-slave-dxdd krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333039, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.3@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:43:59.696374 9925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:59.679261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:47991 (local address 127.23.100.62:38169)
0925 10:43:59.679429 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 10:43:59.679433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:43:59.680484 (+ 1051us) server_negotiation.cc:407] Connection header received
0925 10:43:59.682329 (+ 1845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:43:59.682336 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:43:59.682420 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:43:59.682530 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:43:59.683999 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:59.684861 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:59.686668 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:59.687351 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:59.687481 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:59.691048 (+ 3567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:43:59.691054 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:43:59.691058 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:43:59.691102 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:43:59.693445 (+ 2343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:43:59.694149 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:43:59.694155 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:43:59.694159 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:43:59.694290 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:43:59.694769 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:43:59.694776 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:43:59.694779 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:43:59.695052 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:43:59.695209 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 10:43:59.695829 (+ 620us) server_negotiation.cc:299] Negotiation successful
0925 10:43:59.696044 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 10:43:59.697292 10192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:59.679684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:38169 (local address 127.23.100.3:47991)
0925 10:43:59.680336 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:59.680374 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 10:43:59.682018 (+ 1644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:43:59.682755 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:43:59.682784 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:43:59.683316 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:43:59.683837 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:59.683853 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:59.684998 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:59.685004 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:59.686531 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:59.686547 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:59.687573 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:59.687580 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:59.687848 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:59.688562 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:43:59.688592 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:43:59.690853 (+ 2261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:43:59.693592 (+ 2739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:43:59.693601 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:43:59.693614 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:43:59.694011 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:43:59.694475 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:43:59.694479 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:43:59.694483 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:43:59.694658 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:43:59.695178 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:43:59.695186 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:43:59.695479 (+ 293us) client_negotiation.cc:769] Sending connection context
0925 10:43:59.695726 (+ 247us) client_negotiation.cc:240] Negotiation successful
0925 10:43:59.696193 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":246,"threads_started":1}
I20260925 10:43:59.698800 10190 heartbeater.cc:347] Connected to a master server at 127.23.100.62:38169
I20260925 10:43:59.699119 10190 heartbeater.cc:464] Registering TS with master...
I20260925 10:43:59.699813 10190 heartbeater.cc:511] Master 127.23.100.62:38169 requested a full tablet report, sending...
I20260925 10:43:59.701134 9734 ts_manager.cc:194] Registered new tserver with Master: 208cd03307d64d96b9e5dcea1bf93f74 (127.23.100.3:38783)
I20260925 10:43:59.701822 9734 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.3@KRBTEST.COM'} at 127.23.100.3:47991
I20260925 10:43:59.712281 23952 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:43:59.717736 9923 heartbeater.cc:503] Master 127.23.100.62:38169 was elected leader, sending a full tablet report...
I20260925 10:43:59.729905 9925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:59.716234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55950 (local address 127.23.100.62:38169)
0925 10:43:59.716423 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 10:43:59.716430 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:43:59.716757 (+ 327us) server_negotiation.cc:407] Connection header received
0925 10:43:59.716992 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:43:59.716998 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:43:59.717067 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:43:59.717169 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:43:59.718997 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:59.719835 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:59.721290 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:59.721774 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:59.721907 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:59.724488 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:43:59.724494 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:43:59.724499 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:43:59.724541 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:43:59.726995 (+ 2454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:43:59.727743 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:43:59.727749 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:43:59.727752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:43:59.727893 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:43:59.728325 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:43:59.728331 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:43:59.728339 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:43:59.728611 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:43:59.728923 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0925 10:43:59.729365 (+ 442us) server_negotiation.cc:299] Negotiation successful
0925 10:43:59.729543 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 10:43:59.733363 9734 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55950:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:43:59.739939 9734 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:43:59.773840 10203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:59.751275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:33183 (local address 127.0.0.1:54506)
0925 10:43:59.761742 (+ 10467us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:59.761795 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 10:43:59.761982 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:43:59.766264 (+ 4282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:43:59.766275 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:43:59.766335 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:43:59.766889 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:59.766910 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:59.768818 (+ 1908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:59.768823 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:59.771153 (+ 2330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:59.771166 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:59.772859 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:59.772866 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:59.773054 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:59.773114 (+ 60us) client_negotiation.cc:769] Sending connection context
0925 10:43:59.773284 (+ 170us) client_negotiation.cc:240] Negotiation successful
0925 10:43:59.773467 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10176,"thread_start_us":10906,"threads_started":1}
I20260925 10:43:59.773869 10206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:59.751966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54506 (local address 127.23.100.1:33183)
0925 10:43:59.765504 (+ 13538us) server_negotiation.cc:206] Beginning negotiation
0925 10:43:59.765515 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:43:59.765549 (+ 34us) server_negotiation.cc:407] Connection header received
0925 10:43:59.765666 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:43:59.765676 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:43:59.765992 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:43:59.767153 (+ 1161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:43:59.767402 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:59.768664 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:59.771345 (+ 2681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:59.772373 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:59.772505 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:59.772739 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0925 10:43:59.773332 (+ 593us) server_negotiation.cc:299] Negotiation successful
0925 10:43:59.773506 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13324,"thread_start_us":13133,"threads_started":1}
I20260925 10:43:59.774847 10202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:59.751318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41466 (local address 127.23.100.2:40277)
0925 10:43:59.767459 (+ 16141us) server_negotiation.cc:206] Beginning negotiation
0925 10:43:59.767467 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:43:59.767489 (+ 22us) server_negotiation.cc:407] Connection header received
0925 10:43:59.767563 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:43:59.767568 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:43:59.767735 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:43:59.767915 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:43:59.768524 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:59.770108 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:59.772510 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:59.773571 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:59.773723 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:59.773927 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 10:43:59.774386 (+ 459us) server_negotiation.cc:299] Negotiation successful
0925 10:43:59.774542 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15979,"thread_start_us":15878,"threads_started":1}
I20260925 10:43:59.774914 10204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:59.751326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.2:40277 (local address 127.0.0.1:41466)
0925 10:43:59.764068 (+ 12742us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:59.764089 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 10:43:59.764197 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:43:59.767931 (+ 3734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:43:59.767936 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:43:59.767959 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:43:59.768326 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:59.768338 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:59.770236 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:59.770246 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:59.772372 (+ 2126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:59.772381 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:59.774059 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:59.774065 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:59.774202 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:59.774225 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 10:43:59.774359 (+ 134us) client_negotiation.cc:240] Negotiation successful
0925 10:43:59.774493 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12600,"mutex_wait_us":3,"thread_start_us":12946,"threads_started":1}
I20260925 10:43:59.775255 10205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:59.751653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.3:38783 (local address 127.0.0.1:48670)
0925 10:43:59.767149 (+ 15496us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:59.767169 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:43:59.767289 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:43:59.768184 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:43:59.768188 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:43:59.768206 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:43:59.768580 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:59.768595 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:59.769954 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:59.769961 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:59.772147 (+ 2186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:59.772161 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:59.774261 (+ 2100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:59.774266 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:59.774452 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:59.774496 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 10:43:59.774582 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 10:43:59.774729 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15355,"thread_start_us":15216,"threads_started":1}
I20260925 10:43:59.775471 10207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:59.752060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48670 (local address 127.23.100.3:38783)
0925 10:43:59.764919 (+ 12859us) server_negotiation.cc:206] Beginning negotiation
0925 10:43:59.764930 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:43:59.767415 (+ 2485us) server_negotiation.cc:407] Connection header received
0925 10:43:59.767512 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:43:59.767520 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:43:59.767735 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:43:59.767915 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:43:59.768789 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:59.769798 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:59.772355 (+ 2557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:59.773762 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:59.773901 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:59.774133 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0925 10:43:59.774994 (+ 861us) server_negotiation.cc:299] Negotiation successful
0925 10:43:59.775141 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12654,"thread_start_us":12501,"threads_started":1}
I20260925 10:43:59.778158 9858 tablet_service.cc:1467] Processing CreateTablet for tablet 14c2f35ff1454ca886f791cf467eb65d (DEFAULT_TABLE table=test-table [id=52c3b49cadf243f38774d2aa13e2f3ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:43:59.778506 9992 tablet_service.cc:1467] Processing CreateTablet for tablet 14c2f35ff1454ca886f791cf467eb65d (DEFAULT_TABLE table=test-table [id=52c3b49cadf243f38774d2aa13e2f3ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:43:59.779050 10125 tablet_service.cc:1467] Processing CreateTablet for tablet 14c2f35ff1454ca886f791cf467eb65d (DEFAULT_TABLE table=test-table [id=52c3b49cadf243f38774d2aa13e2f3ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:43:59.779580 9858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14c2f35ff1454ca886f791cf467eb65d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:43:59.779891 9992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14c2f35ff1454ca886f791cf467eb65d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:43:59.780388 10125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14c2f35ff1454ca886f791cf467eb65d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:43:59.790606 10208 tablet_bootstrap.cc:492] T 14c2f35ff1454ca886f791cf467eb65d P 7938af616b92449ab7ce783ce3cf1857: Bootstrap starting.
I20260925 10:43:59.793946 10209 tablet_bootstrap.cc:492] T 14c2f35ff1454ca886f791cf467eb65d P 9f82eaf4bcef48cbba3320263e7dbee9: Bootstrap starting.
I20260925 10:43:59.794332 10208 tablet_bootstrap.cc:654] T 14c2f35ff1454ca886f791cf467eb65d P 7938af616b92449ab7ce783ce3cf1857: Neither blocks nor log segments found. Creating new log.
I20260925 10:43:59.794855 10210 tablet_bootstrap.cc:492] T 14c2f35ff1454ca886f791cf467eb65d P 208cd03307d64d96b9e5dcea1bf93f74: Bootstrap starting.
I20260925 10:43:59.796151 10208 log.cc:824] T 14c2f35ff1454ca886f791cf467eb65d P 7938af616b92449ab7ce783ce3cf1857: Log is configured to *not* fsync() on all Append() calls
I20260925 10:43:59.797281 10209 tablet_bootstrap.cc:654] T 14c2f35ff1454ca886f791cf467eb65d P 9f82eaf4bcef48cbba3320263e7dbee9: Neither blocks nor log segments found. Creating new log.
I20260925 10:43:59.797716 10210 tablet_bootstrap.cc:654] T 14c2f35ff1454ca886f791cf467eb65d P 208cd03307d64d96b9e5dcea1bf93f74: Neither blocks nor log segments found. Creating new log.
I20260925 10:43:59.798518 10209 log.cc:824] T 14c2f35ff1454ca886f791cf467eb65d P 9f82eaf4bcef48cbba3320263e7dbee9: Log is configured to *not* fsync() on all Append() calls
I20260925 10:43:59.798717 10210 log.cc:824] T 14c2f35ff1454ca886f791cf467eb65d P 208cd03307d64d96b9e5dcea1bf93f74: Log is configured to *not* fsync() on all Append() calls
I20260925 10:43:59.799446 10208 tablet_bootstrap.cc:492] T 14c2f35ff1454ca886f791cf467eb65d P 7938af616b92449ab7ce783ce3cf1857: No bootstrap required, opened a new log
I20260925 10:43:59.799726 10208 ts_tablet_manager.cc:1397] T 14c2f35ff1454ca886f791cf467eb65d P 7938af616b92449ab7ce783ce3cf1857: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20260925 10:43:59.801249 10210 tablet_bootstrap.cc:492] T 14c2f35ff1454ca886f791cf467eb65d P 208cd03307d64d96b9e5dcea1bf93f74: No bootstrap required, opened a new log
I20260925 10:43:59.801483 10210 ts_tablet_manager.cc:1397] T 14c2f35ff1454ca886f791cf467eb65d P 208cd03307d64d96b9e5dcea1bf93f74: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.002s
I20260925 10:43:59.801781 10209 tablet_bootstrap.cc:492] T 14c2f35ff1454ca886f791cf467eb65d P 9f82eaf4bcef48cbba3320263e7dbee9: No bootstrap required, opened a new log
I20260925 10:43:59.802023 10209 ts_tablet_manager.cc:1397] T 14c2f35ff1454ca886f791cf467eb65d P 9f82eaf4bcef48cbba3320263e7dbee9: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260925 10:43:59.808451 10210 raft_consensus.cc:348] T 14c2f35ff1454ca886f791cf467eb65d P 208cd03307d64d96b9e5dcea1bf93f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f82eaf4bcef48cbba3320263e7dbee9" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 40277 } } peers { permanent_uuid: "7938af616b92449ab7ce783ce3cf1857" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 33183 } } peers { permanent_uuid: "208cd03307d64d96b9e5dcea1bf93f74" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 38783 } }
I20260925 10:43:59.808805 10210 raft_consensus.cc:374] T 14c2f35ff1454ca886f791cf467eb65d P 208cd03307d64d96b9e5dcea1bf93f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:43:59.808890 10210 raft_consensus.cc:729] T 14c2f35ff1454ca886f791cf467eb65d P 208cd03307d64d96b9e5dcea1bf93f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 208cd03307d64d96b9e5dcea1bf93f74, State: Initialized, Role: FOLLOWER
I20260925 10:43:59.809448 10210 consensus_queue.cc:260] T 14c2f35ff1454ca886f791cf467eb65d P 208cd03307d64d96b9e5dcea1bf93f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f82eaf4bcef48cbba3320263e7dbee9" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 40277 } } peers { permanent_uuid: "7938af616b92449ab7ce783ce3cf1857" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 33183 } } peers { permanent_uuid: "208cd03307d64d96b9e5dcea1bf93f74" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 38783 } }
I20260925 10:43:59.810133 10208 raft_consensus.cc:348] T 14c2f35ff1454ca886f791cf467eb65d P 7938af616b92449ab7ce783ce3cf1857 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f82eaf4bcef48cbba3320263e7dbee9" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 40277 } } peers { permanent_uuid: "7938af616b92449ab7ce783ce3cf1857" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 33183 } } peers { permanent_uuid: "208cd03307d64d96b9e5dcea1bf93f74" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 38783 } }
I20260925 10:43:59.810458 10190 heartbeater.cc:503] Master 127.23.100.62:38169 was elected leader, sending a full tablet report...
I20260925 10:43:59.810540 10208 raft_consensus.cc:374] T 14c2f35ff1454ca886f791cf467eb65d P 7938af616b92449ab7ce783ce3cf1857 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:43:59.810659 10208 raft_consensus.cc:729] T 14c2f35ff1454ca886f791cf467eb65d P 7938af616b92449ab7ce783ce3cf1857 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7938af616b92449ab7ce783ce3cf1857, State: Initialized, Role: FOLLOWER
I20260925 10:43:59.811249 10208 consensus_queue.cc:260] T 14c2f35ff1454ca886f791cf467eb65d P 7938af616b92449ab7ce783ce3cf1857 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f82eaf4bcef48cbba3320263e7dbee9" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 40277 } } peers { permanent_uuid: "7938af616b92449ab7ce783ce3cf1857" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 33183 } } peers { permanent_uuid: "208cd03307d64d96b9e5dcea1bf93f74" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 38783 } }
I20260925 10:43:59.811453 10210 ts_tablet_manager.cc:1428] T 14c2f35ff1454ca886f791cf467eb65d P 208cd03307d64d96b9e5dcea1bf93f74: Time spent starting tablet: real 0.010s user 0.006s sys 0.005s
I20260925 10:43:59.812186 10209 raft_consensus.cc:348] T 14c2f35ff1454ca886f791cf467eb65d P 9f82eaf4bcef48cbba3320263e7dbee9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f82eaf4bcef48cbba3320263e7dbee9" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 40277 } } peers { permanent_uuid: "7938af616b92449ab7ce783ce3cf1857" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 33183 } } peers { permanent_uuid: "208cd03307d64d96b9e5dcea1bf93f74" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 38783 } }
I20260925 10:43:59.812593 10209 raft_consensus.cc:374] T 14c2f35ff1454ca886f791cf467eb65d P 9f82eaf4bcef48cbba3320263e7dbee9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:43:59.812685 10209 raft_consensus.cc:729] T 14c2f35ff1454ca886f791cf467eb65d P 9f82eaf4bcef48cbba3320263e7dbee9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f82eaf4bcef48cbba3320263e7dbee9, State: Initialized, Role: FOLLOWER
I20260925 10:43:59.812965 10208 ts_tablet_manager.cc:1428] T 14c2f35ff1454ca886f791cf467eb65d P 7938af616b92449ab7ce783ce3cf1857: Time spent starting tablet: real 0.013s user 0.007s sys 0.006s
I20260925 10:43:59.813383 10209 consensus_queue.cc:260] T 14c2f35ff1454ca886f791cf467eb65d P 9f82eaf4bcef48cbba3320263e7dbee9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f82eaf4bcef48cbba3320263e7dbee9" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 40277 } } peers { permanent_uuid: "7938af616b92449ab7ce783ce3cf1857" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 33183 } } peers { permanent_uuid: "208cd03307d64d96b9e5dcea1bf93f74" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 38783 } }
I20260925 10:43:59.814860 10057 heartbeater.cc:503] Master 127.23.100.62:38169 was elected leader, sending a full tablet report...
I20260925 10:43:59.815454 10209 ts_tablet_manager.cc:1428] T 14c2f35ff1454ca886f791cf467eb65d P 9f82eaf4bcef48cbba3320263e7dbee9: Time spent starting tablet: real 0.013s user 0.005s sys 0.009s
W20260925 10:43:59.928138 10058 tablet.cc:2369] T 14c2f35ff1454ca886f791cf467eb65d P 9f82eaf4bcef48cbba3320263e7dbee9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:43:59.928205 10191 tablet.cc:2369] T 14c2f35ff1454ca886f791cf467eb65d P 208cd03307d64d96b9e5dcea1bf93f74: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:43:59.937261 9924 tablet.cc:2369] T 14c2f35ff1454ca886f791cf467eb65d P 7938af616b92449ab7ce783ce3cf1857: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:43:59.955260 10214 raft_consensus.cc:482] T 14c2f35ff1454ca886f791cf467eb65d P 208cd03307d64d96b9e5dcea1bf93f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:43:59.955571 10214 raft_consensus.cc:504] T 14c2f35ff1454ca886f791cf467eb65d P 208cd03307d64d96b9e5dcea1bf93f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f82eaf4bcef48cbba3320263e7dbee9" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 40277 } } peers { permanent_uuid: "7938af616b92449ab7ce783ce3cf1857" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 33183 } } peers { permanent_uuid: "208cd03307d64d96b9e5dcea1bf93f74" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 38783 } }
I20260925 10:43:59.957319 10214 leader_election.cc:288] T 14c2f35ff1454ca886f791cf467eb65d P 208cd03307d64d96b9e5dcea1bf93f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f82eaf4bcef48cbba3320263e7dbee9 (127.23.100.2:40277), 7938af616b92449ab7ce783ce3cf1857 (127.23.100.1:33183)
I20260925 10:43:59.963145 10202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:59.957781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:46585 (local address 127.23.100.2:40277)
0925 10:43:59.958055 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0925 10:43:59.958062 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:43:59.958382 (+ 320us) server_negotiation.cc:407] Connection header received
0925 10:43:59.958622 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:43:59.958626 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:43:59.958686 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:43:59.958802 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:43:59.959231 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:59.959932 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:59.961529 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:59.962295 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:59.962474 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:59.962542 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 10:43:59.962797 (+ 255us) server_negotiation.cc:299] Negotiation successful
0925 10:43:59.962913 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 10:43:59.963197 10192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:59.957702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.2:40277 (local address 127.23.100.3:46585)
0925 10:43:59.958269 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:59.958283 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:43:59.958459 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:43:59.958822 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:43:59.958826 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:43:59.958919 (+ 93us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:43:59.959086 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:59.959101 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:59.960063 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:59.960070 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:59.961382 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:59.961393 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:59.962466 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:59.962471 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:59.962648 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:59.962670 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 10:43:59.962772 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 10:43:59.962896 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"spinlock_wait_cycles":23296}
I20260925 10:43:59.963699 10206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:59.957858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:50445 (local address 127.23.100.1:33183)
0925 10:43:59.958122 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0925 10:43:59.958127 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:43:59.958822 (+ 695us) server_negotiation.cc:407] Connection header received
0925 10:43:59.959023 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:43:59.959033 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:43:59.959093 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:43:59.959188 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:43:59.959706 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:59.960381 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:59.962012 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:59.962812 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:59.962981 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:59.963046 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 10:43:59.963385 (+ 339us) server_negotiation.cc:299] Negotiation successful
0925 10:43:59.963506 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260925 10:43:59.964272 10217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:59.957916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:33183 (local address 127.23.100.3:50445)
0925 10:43:59.958676 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0925 10:43:59.958691 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:43:59.958870 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:43:59.959209 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:43:59.959213 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:43:59.959237 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:43:59.959535 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:59.959543 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:59.960527 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:59.960532 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:59.961857 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:43:59.961868 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:43:59.962999 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:43:59.963004 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:43:59.963155 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:59.963186 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 10:43:59.963874 (+ 688us) client_negotiation.cc:240] Negotiation successful
0925 10:43:59.964003 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"mutex_wait_us":1,"thread_start_us":409,"threads_started":1}
I20260925 10:43:59.964196 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14c2f35ff1454ca886f791cf467eb65d" candidate_uuid: "208cd03307d64d96b9e5dcea1bf93f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f82eaf4bcef48cbba3320263e7dbee9" is_pre_election: true
I20260925 10:43:59.964761 10012 raft_consensus.cc:2454] T 14c2f35ff1454ca886f791cf467eb65d P 9f82eaf4bcef48cbba3320263e7dbee9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 208cd03307d64d96b9e5dcea1bf93f74 in term 0.
I20260925 10:43:59.965286 9878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14c2f35ff1454ca886f791cf467eb65d" candidate_uuid: "208cd03307d64d96b9e5dcea1bf93f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7938af616b92449ab7ce783ce3cf1857" is_pre_election: true
I20260925 10:43:59.965584 10081 leader_election.cc:302] T 14c2f35ff1454ca886f791cf467eb65d P 208cd03307d64d96b9e5dcea1bf93f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 208cd03307d64d96b9e5dcea1bf93f74, 9f82eaf4bcef48cbba3320263e7dbee9; no voters:
I20260925 10:43:59.965857 9878 raft_consensus.cc:2454] T 14c2f35ff1454ca886f791cf467eb65d P 7938af616b92449ab7ce783ce3cf1857 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 208cd03307d64d96b9e5dcea1bf93f74 in term 0.
I20260925 10:43:59.965965 10214 raft_consensus.cc:2781] T 14c2f35ff1454ca886f791cf467eb65d P 208cd03307d64d96b9e5dcea1bf93f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:43:59.966120 10214 raft_consensus.cc:482] T 14c2f35ff1454ca886f791cf467eb65d P 208cd03307d64d96b9e5dcea1bf93f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:43:59.966228 10214 raft_consensus.cc:3037] T 14c2f35ff1454ca886f791cf467eb65d P 208cd03307d64d96b9e5dcea1bf93f74 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:43:59.968356 10214 raft_consensus.cc:504] T 14c2f35ff1454ca886f791cf467eb65d P 208cd03307d64d96b9e5dcea1bf93f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f82eaf4bcef48cbba3320263e7dbee9" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 40277 } } peers { permanent_uuid: "7938af616b92449ab7ce783ce3cf1857" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 33183 } } peers { permanent_uuid: "208cd03307d64d96b9e5dcea1bf93f74" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 38783 } }
I20260925 10:43:59.968972 10214 leader_election.cc:288] T 14c2f35ff1454ca886f791cf467eb65d P 208cd03307d64d96b9e5dcea1bf93f74 [CANDIDATE]: Term 1 election: Requested vote from peers 9f82eaf4bcef48cbba3320263e7dbee9 (127.23.100.2:40277), 7938af616b92449ab7ce783ce3cf1857 (127.23.100.1:33183)
I20260925 10:43:59.969318 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14c2f35ff1454ca886f791cf467eb65d" candidate_uuid: "208cd03307d64d96b9e5dcea1bf93f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f82eaf4bcef48cbba3320263e7dbee9"
I20260925 10:43:59.969372 9878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14c2f35ff1454ca886f791cf467eb65d" candidate_uuid: "208cd03307d64d96b9e5dcea1bf93f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7938af616b92449ab7ce783ce3cf1857"
I20260925 10:43:59.969502 10012 raft_consensus.cc:3037] T 14c2f35ff1454ca886f791cf467eb65d P 9f82eaf4bcef48cbba3320263e7dbee9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:43:59.969579 9878 raft_consensus.cc:3037] T 14c2f35ff1454ca886f791cf467eb65d P 7938af616b92449ab7ce783ce3cf1857 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:43:59.971751 10012 raft_consensus.cc:2454] T 14c2f35ff1454ca886f791cf467eb65d P 9f82eaf4bcef48cbba3320263e7dbee9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 208cd03307d64d96b9e5dcea1bf93f74 in term 1.
I20260925 10:43:59.971871 9878 raft_consensus.cc:2454] T 14c2f35ff1454ca886f791cf467eb65d P 7938af616b92449ab7ce783ce3cf1857 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 208cd03307d64d96b9e5dcea1bf93f74 in term 1.
I20260925 10:43:59.972143 10081 leader_election.cc:302] T 14c2f35ff1454ca886f791cf467eb65d P 208cd03307d64d96b9e5dcea1bf93f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 208cd03307d64d96b9e5dcea1bf93f74, 9f82eaf4bcef48cbba3320263e7dbee9; no voters:
I20260925 10:43:59.972438 10214 raft_consensus.cc:2781] T 14c2f35ff1454ca886f791cf467eb65d P 208cd03307d64d96b9e5dcea1bf93f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:43:59.972846 10214 raft_consensus.cc:686] T 14c2f35ff1454ca886f791cf467eb65d P 208cd03307d64d96b9e5dcea1bf93f74 [term 1 LEADER]: Becoming Leader. State: Replica: 208cd03307d64d96b9e5dcea1bf93f74, State: Running, Role: LEADER
I20260925 10:43:59.973294 10214 consensus_queue.cc:237] T 14c2f35ff1454ca886f791cf467eb65d P 208cd03307d64d96b9e5dcea1bf93f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f82eaf4bcef48cbba3320263e7dbee9" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 40277 } } peers { permanent_uuid: "7938af616b92449ab7ce783ce3cf1857" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 33183 } } peers { permanent_uuid: "208cd03307d64d96b9e5dcea1bf93f74" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 38783 } }
I20260925 10:43:59.977519 9734 catalog_manager.cc:5495] T 14c2f35ff1454ca886f791cf467eb65d P 208cd03307d64d96b9e5dcea1bf93f74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 208cd03307d64d96b9e5dcea1bf93f74 (127.23.100.3). New cstate: current_term: 1 leader_uuid: "208cd03307d64d96b9e5dcea1bf93f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f82eaf4bcef48cbba3320263e7dbee9" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 40277 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7938af616b92449ab7ce783ce3cf1857" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 33183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "208cd03307d64d96b9e5dcea1bf93f74" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 38783 } health_report { overall_health: HEALTHY } } }
I20260925 10:44:00.000366 10207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:43:59.994871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48686 (local address 127.23.100.3:38783)
0925 10:43:59.995062 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 10:43:59.995071 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:43:59.995100 (+ 29us) server_negotiation.cc:407] Connection header received
0925 10:43:59.995332 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:43:59.995336 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:43:59.995406 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:43:59.995554 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:43:59.995996 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:59.996719 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:59.998129 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:43:59.998565 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:43:59.998700 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:43:59.999305 (+ 605us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:43:59.999780 (+ 475us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:43:59.999962 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:00.000097 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 10:44:00.000143 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 10:44:00.007968 9878 raft_consensus.cc:1261] T 14c2f35ff1454ca886f791cf467eb65d P 7938af616b92449ab7ce783ce3cf1857 [term 1 FOLLOWER]: Refusing update from remote peer 208cd03307d64d96b9e5dcea1bf93f74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:44:00.008040 10012 raft_consensus.cc:1261] T 14c2f35ff1454ca886f791cf467eb65d P 9f82eaf4bcef48cbba3320263e7dbee9 [term 1 FOLLOWER]: Refusing update from remote peer 208cd03307d64d96b9e5dcea1bf93f74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:44:00.008967 10218 consensus_queue.cc:1035] T 14c2f35ff1454ca886f791cf467eb65d P 208cd03307d64d96b9e5dcea1bf93f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7938af616b92449ab7ce783ce3cf1857" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:44:00.009297 10214 consensus_queue.cc:1035] T 14c2f35ff1454ca886f791cf467eb65d P 208cd03307d64d96b9e5dcea1bf93f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f82eaf4bcef48cbba3320263e7dbee9" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 40277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:44:00.020511 10221 mvcc.cc:205] Tried to move back new op lower bound from 7333204131863470080 to 7333204131734118400. Current Snapshot: MvccSnapshot[applied={T|T < 7333204131863470080}]
I20260925 10:44:00.035506 9732 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:44:00.035830 9732 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:44:00.039592 9732 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7fc3832077344c988678da316c5250e9: Sending DeleteTablet for 3 replicas of tablet 14c2f35ff1454ca886f791cf467eb65d
I20260925 10:44:00.040632 9992 tablet_service.cc:1514] Processing DeleteTablet for tablet 14c2f35ff1454ca886f791cf467eb65d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:44:00 UTC) from {username='kudu', principal='kudu/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:41466
I20260925 10:44:00.040678 10125 tablet_service.cc:1514] Processing DeleteTablet for tablet 14c2f35ff1454ca886f791cf467eb65d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:44:00 UTC) from {username='kudu', principal='kudu/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:48670
I20260925 10:44:00.040668 9858 tablet_service.cc:1514] Processing DeleteTablet for tablet 14c2f35ff1454ca886f791cf467eb65d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:44:00 UTC) from {username='kudu', principal='kudu/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:54506
I20260925 10:44:00.041522 10231 tablet_replica.cc:330] T 14c2f35ff1454ca886f791cf467eb65d P 9f82eaf4bcef48cbba3320263e7dbee9: stopping tablet replica
I20260925 10:44:00.041939 10231 raft_consensus.cc:2229] T 14c2f35ff1454ca886f791cf467eb65d P 9f82eaf4bcef48cbba3320263e7dbee9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:44:00.042536 10231 raft_consensus.cc:2258] T 14c2f35ff1454ca886f791cf467eb65d P 9f82eaf4bcef48cbba3320263e7dbee9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:44:00.046406 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 9795
I20260925 10:44:00.052946 10232 tablet_replica.cc:330] T 14c2f35ff1454ca886f791cf467eb65d P 208cd03307d64d96b9e5dcea1bf93f74: stopping tablet replica
I20260925 10:44:00.053169 10233 tablet_replica.cc:330] T 14c2f35ff1454ca886f791cf467eb65d P 7938af616b92449ab7ce783ce3cf1857: stopping tablet replica
I20260925 10:44:00.053452 10232 raft_consensus.cc:2229] T 14c2f35ff1454ca886f791cf467eb65d P 208cd03307d64d96b9e5dcea1bf93f74 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:44:00.053699 10233 raft_consensus.cc:2229] T 14c2f35ff1454ca886f791cf467eb65d P 7938af616b92449ab7ce783ce3cf1857 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:44:00.054225 10233 raft_consensus.cc:2258] T 14c2f35ff1454ca886f791cf467eb65d P 7938af616b92449ab7ce783ce3cf1857 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:44:00.054215 10232 raft_consensus.cc:2258] T 14c2f35ff1454ca886f791cf467eb65d P 208cd03307d64d96b9e5dcea1bf93f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:44:00.056454 10233 ts_tablet_manager.cc:1905] T 14c2f35ff1454ca886f791cf467eb65d P 7938af616b92449ab7ce783ce3cf1857: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:44:00.056700 10232 ts_tablet_manager.cc:1905] T 14c2f35ff1454ca886f791cf467eb65d P 208cd03307d64d96b9e5dcea1bf93f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:44:00.057768 10231 ts_tablet_manager.cc:1905] T 14c2f35ff1454ca886f791cf467eb65d P 9f82eaf4bcef48cbba3320263e7dbee9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:44:00.063463 10233 ts_tablet_manager.cc:1918] T 14c2f35ff1454ca886f791cf467eb65d P 7938af616b92449ab7ce783ce3cf1857: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:44:00.063665 10233 log.cc:1196] T 14c2f35ff1454ca886f791cf467eb65d P 7938af616b92449ab7ce783ce3cf1857: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/ts-0/wal/wals/14c2f35ff1454ca886f791cf467eb65d
I20260925 10:44:00.064260 10233 ts_tablet_manager.cc:1939] T 14c2f35ff1454ca886f791cf467eb65d P 7938af616b92449ab7ce783ce3cf1857: Deleting consensus metadata
I20260925 10:44:00.064355 10232 ts_tablet_manager.cc:1918] T 14c2f35ff1454ca886f791cf467eb65d P 208cd03307d64d96b9e5dcea1bf93f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:44:00.064513 10232 log.cc:1196] T 14c2f35ff1454ca886f791cf467eb65d P 208cd03307d64d96b9e5dcea1bf93f74: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/ts-2/wal/wals/14c2f35ff1454ca886f791cf467eb65d
I20260925 10:44:00.065042 10232 ts_tablet_manager.cc:1939] T 14c2f35ff1454ca886f791cf467eb65d P 208cd03307d64d96b9e5dcea1bf93f74: Deleting consensus metadata
I20260925 10:44:00.066277 9718 catalog_manager.cc:4841] TS 7938af616b92449ab7ce783ce3cf1857 (127.23.100.1:33183): tablet 14c2f35ff1454ca886f791cf467eb65d (table test-table [id=52c3b49cadf243f38774d2aa13e2f3ee]) successfully deleted
I20260925 10:44:00.066881 9720 catalog_manager.cc:4841] TS 208cd03307d64d96b9e5dcea1bf93f74 (127.23.100.3:38783): tablet 14c2f35ff1454ca886f791cf467eb65d (table test-table [id=52c3b49cadf243f38774d2aa13e2f3ee]) successfully deleted
I20260925 10:44:00.069584 10231 ts_tablet_manager.cc:1918] T 14c2f35ff1454ca886f791cf467eb65d P 9f82eaf4bcef48cbba3320263e7dbee9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:44:00.069747 10231 log.cc:1196] T 14c2f35ff1454ca886f791cf467eb65d P 9f82eaf4bcef48cbba3320263e7dbee9: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790332740062128-23952-0/minicluster-data/ts-1/wal/wals/14c2f35ff1454ca886f791cf467eb65d
I20260925 10:44:00.070254 10231 ts_tablet_manager.cc:1939] T 14c2f35ff1454ca886f791cf467eb65d P 9f82eaf4bcef48cbba3320263e7dbee9: Deleting consensus metadata
I20260925 10:44:00.071936 9721 catalog_manager.cc:4841] TS 9f82eaf4bcef48cbba3320263e7dbee9 (127.23.100.2:40277): tablet 14c2f35ff1454ca886f791cf467eb65d (table test-table [id=52c3b49cadf243f38774d2aa13e2f3ee]) successfully deleted
Sep 25 10:44:00 dist-test-slave-dxdd krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.1@KRBTEST.COM
I20260925 10:44:00.080665 10206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:00.048252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54518 (local address 127.23.100.1:33183)
0925 10:44:00.048473 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:00.048482 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:00.064787 (+ 16305us) server_negotiation.cc:407] Connection header received
0925 10:44:00.065263 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:00.065271 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:00.065364 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:00.065442 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:00.067320 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:00.068073 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:00.069416 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:00.069998 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:00.070148 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:00.074905 (+ 4757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:00.074916 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:00.074929 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:44:00.074959 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:00.077841 (+ 2882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:00.078593 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:00.078600 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:00.078602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:00.078737 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:00.079126 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:00.079135 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:00.079137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:00.079456 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:00.079774 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:00.080056 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 10:44:00.080295 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 10:44:00.081364 9918 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:44:00.170245 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 9795
I20260925 10:44:00.188493 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 9929
Sep 25 10:44:00 dist-test-slave-dxdd krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.2@KRBTEST.COM
I20260925 10:44:00.204710 10202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:00.189252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41476 (local address 127.23.100.2:40277)
0925 10:44:00.189431 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:00.189438 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:00.189581 (+ 143us) server_negotiation.cc:407] Connection header received
0925 10:44:00.189793 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:00.189799 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:00.189886 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:00.189998 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:00.191402 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:00.192177 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:00.193923 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:00.194362 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:00.194491 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:00.198726 (+ 4235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:00.198737 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:00.198781 (+ 44us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:44:00.198809 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:00.201927 (+ 3118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:00.202623 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:00.202632 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:00.202635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:00.202766 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:00.203131 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:00.203135 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:00.203139 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:00.203539 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:00.203809 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:00.204143 (+ 334us) server_negotiation.cc:299] Negotiation successful
0925 10:44:00.204351 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 10:44:00.205312 10052 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:44:00.299355 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 9929
I20260925 10:44:00.322835 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 10062
Sep 25 10:44:00 dist-test-slave-dxdd krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.3@KRBTEST.COM
I20260925 10:44:00.339082 10207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:00.323645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48688 (local address 127.23.100.3:38783)
0925 10:44:00.323820 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:00.323824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:00.324035 (+ 211us) server_negotiation.cc:407] Connection header received
0925 10:44:00.324259 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:00.324268 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:00.324329 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:00.324449 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:00.325777 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:00.326667 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:00.327986 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:00.328438 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:00.328571 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:00.333015 (+ 4444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:00.333025 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:00.333039 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:44:00.333074 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:00.336109 (+ 3035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:00.336771 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:00.336786 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:00.336788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:00.336929 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:00.337312 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:00.337322 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:00.337332 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:00.337670 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:00.338035 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:00.338267 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 10:44:00.338778 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 10:44:00.339774 10185 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:44:00.457604 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 10062
I20260925 10:44:00.476876 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 9702
I20260925 10:44:00.477938 9764 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:44:00.586366 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 9702
2026-09-25T10:44:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4252 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T10:44:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:44:00Z Disabled control of system clock
I20260925 10:44:00.659192 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:43213
--webserver_interface=127.23.100.62
--webserver_port=0
--builtin_ntp_servers=127.23.100.20:44801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.62:43213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 10:44:00.976054 10242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:44:00.976326 10242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:44:00.976472 10242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:44:00.984161 10242 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:44:00.984287 10242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:44:00.984359 10242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:44:00.984472 10242 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:44:00.984557 10242 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:44:00.993571 10242 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:44801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.62:43213
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:43213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:44:00.994875 10242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:44:00.996562 10242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:44:01.007838 10247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:01.007932 10250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:01.007869 10248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:44:01.008864 10242 server_base.cc:1034] running on GCE node
I20260925 10:44:01.009625 10242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:44:01.011039 10242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:44:01.012269 10242 hybrid_clock.cc:648] HybridClock initialized: now 1790333041012235 us; error 46 us; skew 500 ppm
I20260925 10:44:01.012605 10242 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:44:01.019747 10242 webserver.cc:460] Webserver started at http://127.23.100.62:38995/ using document root <none> and password file <none>
I20260925 10:44:01.020416 10242 fs_manager.cc:359] Metadata directory not provided
I20260925 10:44:01.020550 10242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:44:01.020876 10242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:44:01.023731 10242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/master-0/data/instance:
uuid: "f8fb0dfd4256457689a9e86a84ed4a5c"
format_stamp: "Formatted at 2026-09-25 10:44:01 on dist-test-slave-dxdd"
server_key: "ddffa42382ecfad3f59ccba2faed6e67"
server_key_iv: "9bd2cfae14b131279bf9deffb240bfe8"
server_key_version: "encryptionkey@0"
I20260925 10:44:01.024510 10242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/master-0/wal/instance:
uuid: "f8fb0dfd4256457689a9e86a84ed4a5c"
format_stamp: "Formatted at 2026-09-25 10:44:01 on dist-test-slave-dxdd"
server_key: "ddffa42382ecfad3f59ccba2faed6e67"
server_key_iv: "9bd2cfae14b131279bf9deffb240bfe8"
server_key_version: "encryptionkey@0"
I20260925 10:44:01.029309 10242 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260925 10:44:01.032488 10256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:44:01.033695 10242 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 10:44:01.033872 10242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/master-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/master-0/wal
uuid: "f8fb0dfd4256457689a9e86a84ed4a5c"
format_stamp: "Formatted at 2026-09-25 10:44:01 on dist-test-slave-dxdd"
server_key: "ddffa42382ecfad3f59ccba2faed6e67"
server_key_iv: "9bd2cfae14b131279bf9deffb240bfe8"
server_key_version: "encryptionkey@0"
I20260925 10:44:01.034076 10242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:44:01.078660 10242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:44:01.079533 10242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:44:01.079829 10242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:44:01.097754 10242 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.62:43213
I20260925 10:44:01.097790 10307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.62:43213 every 8 connection(s)
I20260925 10:44:01.099138 10242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
I20260925 10:44:01.102692 10308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:44:01.109157 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 10242
I20260925 10:44:01.109361 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/master-0/wal/instance
I20260925 10:44:01.109762 23952 external_mini_cluster.cc:1428] Setting key f7d58e09a8c6d0f9dfb6e188d0c7444d
I20260925 10:44:01.113145 10308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8fb0dfd4256457689a9e86a84ed4a5c: Bootstrap starting.
I20260925 10:44:01.116343 10308 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8fb0dfd4256457689a9e86a84ed4a5c: Neither blocks nor log segments found. Creating new log.
I20260925 10:44:01.117457 10308 log.cc:824] T 00000000000000000000000000000000 P f8fb0dfd4256457689a9e86a84ed4a5c: Log is configured to *not* fsync() on all Append() calls
I20260925 10:44:01.120404 10308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8fb0dfd4256457689a9e86a84ed4a5c: No bootstrap required, opened a new log
I20260925 10:44:01.122923 10311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:01.111614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39218 (local address 127.23.100.62:43213)
0925 10:44:01.112557 (+ 943us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:01.112567 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:01.112603 (+ 36us) server_negotiation.cc:407] Connection header received
0925 10:44:01.114285 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:01.114369 (+ 84us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:01.114805 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:01.115096 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:01.115688 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:01.116928 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:01.118808 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:01.119442 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:01.119575 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:01.120770 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:01.120777 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:01.120791 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:44:01.121303 (+ 512us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:01.121368 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:44:01.121386 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:01.121472 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:01.121703 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 10:44:01.122000 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":342,"threads_started":1}
I20260925 10:44:01.128170 10308 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8fb0dfd4256457689a9e86a84ed4a5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8fb0dfd4256457689a9e86a84ed4a5c" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 43213 } }
I20260925 10:44:01.128532 10308 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8fb0dfd4256457689a9e86a84ed4a5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:44:01.128651 10308 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8fb0dfd4256457689a9e86a84ed4a5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8fb0dfd4256457689a9e86a84ed4a5c, State: Initialized, Role: FOLLOWER
I20260925 10:44:01.129205 10308 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8fb0dfd4256457689a9e86a84ed4a5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8fb0dfd4256457689a9e86a84ed4a5c" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 43213 } }
I20260925 10:44:01.129433 10308 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8fb0dfd4256457689a9e86a84ed4a5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:44:01.129593 10308 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8fb0dfd4256457689a9e86a84ed4a5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:44:01.129756 10308 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8fb0dfd4256457689a9e86a84ed4a5c [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:44:01.131868 10308 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8fb0dfd4256457689a9e86a84ed4a5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8fb0dfd4256457689a9e86a84ed4a5c" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 43213 } }
I20260925 10:44:01.132328 10308 leader_election.cc:302] T 00000000000000000000000000000000 P f8fb0dfd4256457689a9e86a84ed4a5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8fb0dfd4256457689a9e86a84ed4a5c; no voters:
I20260925 10:44:01.132869 10308 leader_election.cc:288] T 00000000000000000000000000000000 P f8fb0dfd4256457689a9e86a84ed4a5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:44:01.133056 10313 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8fb0dfd4256457689a9e86a84ed4a5c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:44:01.134297 10313 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8fb0dfd4256457689a9e86a84ed4a5c [term 1 LEADER]: Becoming Leader. State: Replica: f8fb0dfd4256457689a9e86a84ed4a5c, State: Running, Role: LEADER
I20260925 10:44:01.134847 10313 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8fb0dfd4256457689a9e86a84ed4a5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8fb0dfd4256457689a9e86a84ed4a5c" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 43213 } }
I20260925 10:44:01.135099 10308 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8fb0dfd4256457689a9e86a84ed4a5c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:44:01.137316 10315 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8fb0dfd4256457689a9e86a84ed4a5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8fb0dfd4256457689a9e86a84ed4a5c. Latest consensus state: current_term: 1 leader_uuid: "f8fb0dfd4256457689a9e86a84ed4a5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8fb0dfd4256457689a9e86a84ed4a5c" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 43213 } } }
I20260925 10:44:01.137706 10315 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8fb0dfd4256457689a9e86a84ed4a5c [sys.catalog]: This master's current role is: LEADER
I20260925 10:44:01.138214 10314 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8fb0dfd4256457689a9e86a84ed4a5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8fb0dfd4256457689a9e86a84ed4a5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8fb0dfd4256457689a9e86a84ed4a5c" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 43213 } } }
I20260925 10:44:01.138506 10314 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8fb0dfd4256457689a9e86a84ed4a5c [sys.catalog]: This master's current role is: LEADER
I20260925 10:44:01.140203 10320 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:44:01.145006 10320 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:44:01.153203 10320 catalog_manager.cc:1347] Generated new cluster ID: eda54c972bd741f29b4a403bbf4d755e
I20260925 10:44:01.153303 10320 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:44:01.165349 10320 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:44:01.166604 10320 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:44:01.172659 10320 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8fb0dfd4256457689a9e86a84ed4a5c: Generated new TSK 0
I20260925 10:44:01.173363 10320 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:44:01.184478 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.1:0
--local_ip_for_outbound_sockets=127.23.100.1
--webserver_interface=127.23.100.1
--webserver_port=0
--tserver_master_addrs=127.23.100.62:43213
--builtin_ntp_servers=127.23.100.20:44801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 10:44:01.527951 10333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:44:01.528259 10333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:44:01.528414 10333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:44:01.535621 10333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:44:01.535732 10333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:44:01.535916 10333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.1
I20260925 10:44:01.545296 10333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:44801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.1
--webserver_port=0
--tserver_master_addrs=127.23.100.62:43213
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.1
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:44:01.546607 10333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:44:01.548317 10333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:44:01.560976 10341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:01.561115 10338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:01.561141 10339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:44:01.561949 10333 server_base.cc:1034] running on GCE node
I20260925 10:44:01.562602 10333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:44:01.563457 10333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:44:01.564682 10333 hybrid_clock.cc:648] HybridClock initialized: now 1790333041564649 us; error 43 us; skew 500 ppm
I20260925 10:44:01.564975 10333 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:44:01.567126 10333 webserver.cc:460] Webserver started at http://127.23.100.1:41791/ using document root <none> and password file <none>
I20260925 10:44:01.567782 10333 fs_manager.cc:359] Metadata directory not provided
I20260925 10:44:01.567925 10333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:44:01.568229 10333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:44:01.571034 10333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/ts-0/data/instance:
uuid: "ae8a12c32675488482d3eb3c171ede2d"
format_stamp: "Formatted at 2026-09-25 10:44:01 on dist-test-slave-dxdd"
server_key: "b78afb26605a27750e96a97337894dc3"
server_key_iv: "93eae09b518d1d2e90346e9748343e86"
server_key_version: "encryptionkey@0"
I20260925 10:44:01.571851 10333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/ts-0/wal/instance:
uuid: "ae8a12c32675488482d3eb3c171ede2d"
format_stamp: "Formatted at 2026-09-25 10:44:01 on dist-test-slave-dxdd"
server_key: "b78afb26605a27750e96a97337894dc3"
server_key_iv: "93eae09b518d1d2e90346e9748343e86"
server_key_version: "encryptionkey@0"
I20260925 10:44:01.576725 10333 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 10:44:01.579955 10347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:44:01.581116 10333 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 10:44:01.581282 10333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/ts-0/wal
uuid: "ae8a12c32675488482d3eb3c171ede2d"
format_stamp: "Formatted at 2026-09-25 10:44:01 on dist-test-slave-dxdd"
server_key: "b78afb26605a27750e96a97337894dc3"
server_key_iv: "93eae09b518d1d2e90346e9748343e86"
server_key_version: "encryptionkey@0"
I20260925 10:44:01.581482 10333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:44:01.595463 10333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:44:01.596231 10333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:44:01.596530 10333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:44:01.597419 10333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:44:01.598834 10333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:44:01.598951 10333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:44:01.599099 10333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:44:01.599189 10333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:44:01.631625 10333 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.1:40689
I20260925 10:44:01.631665 10459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.1:40689 every 8 connection(s)
I20260925 10:44:01.633049 10333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
I20260925 10:44:01.635632 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 10333
I20260925 10:44:01.635830 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/ts-0/wal/instance
I20260925 10:44:01.636385 23952 external_mini_cluster.cc:1428] Setting key 9da0d10c4a700d5f24bc83591da367e9
I20260925 10:44:01.639712 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.2:0
--local_ip_for_outbound_sockets=127.23.100.2
--webserver_interface=127.23.100.2
--webserver_port=0
--tserver_master_addrs=127.23.100.62:43213
--builtin_ntp_servers=127.23.100.20:44801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 10:44:01.646785 10462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:01.635685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:43007 (local address 127.23.100.62:43213)
0925 10:44:01.636237 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:01.636242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:01.637649 (+ 1407us) server_negotiation.cc:407] Connection header received
0925 10:44:01.640058 (+ 2409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:01.640066 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:01.640186 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:01.640306 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:01.641377 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:01.642515 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:01.644058 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:01.644452 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:01.644602 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:01.645917 (+ 1315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:01.645928 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:01.645931 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:44:01.645947 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:01.645986 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:44:01.645998 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:01.646089 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:01.646324 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 10:44:01.646420 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":275,"threads_started":1}
I20260925 10:44:01.647651 10463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:01.636332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:43213 (local address 127.23.100.1:43007)
0925 10:44:01.637417 (+ 1085us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:01.637495 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0925 10:44:01.639642 (+ 2147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:44:01.640572 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:44:01.640579 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:44:01.640662 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:44:01.641206 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:01.641222 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:01.642663 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:01.642667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:01.643912 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:01.643923 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:01.644699 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:01.644708 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:01.644942 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:01.645620 (+ 678us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:44:01.645622 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:44:01.645658 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:44:01.645733 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:44:01.646104 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:44:01.646117 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:44:01.646128 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 10:44:01.646294 (+ 166us) client_negotiation.cc:240] Negotiation successful
0925 10:44:01.646679 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":294,"threads_started":1}
I20260925 10:44:01.649087 10460 heartbeater.cc:347] Connected to a master server at 127.23.100.62:43213
I20260925 10:44:01.649356 10460 heartbeater.cc:464] Registering TS with master...
I20260925 10:44:01.650045 10460 heartbeater.cc:511] Master 127.23.100.62:43213 requested a full tablet report, sending...
I20260925 10:44:01.651973 10273 ts_manager.cc:194] Registered new tserver with Master: ae8a12c32675488482d3eb3c171ede2d (127.23.100.1:40689)
I20260925 10:44:01.653353 10273 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.100.1:43007
W20260925 10:44:01.959407 10464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:44:01.959647 10464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:44:01.959797 10464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:44:01.966598 10464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:44:01.966676 10464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:44:01.966892 10464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.2
I20260925 10:44:01.975682 10464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:44801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.2
--webserver_port=0
--tserver_master_addrs=127.23.100.62:43213
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.2
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:44:01.976871 10464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:44:01.978539 10464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:44:01.991761 10469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:01.991737 10472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:01.991679 10470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:44:01.992228 10464 server_base.cc:1034] running on GCE node
I20260925 10:44:01.992893 10464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:44:01.993762 10464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:44:01.994978 10464 hybrid_clock.cc:648] HybridClock initialized: now 1790333041994935 us; error 57 us; skew 500 ppm
I20260925 10:44:01.995270 10464 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:44:01.997454 10464 webserver.cc:460] Webserver started at http://127.23.100.2:44253/ using document root <none> and password file <none>
I20260925 10:44:01.998102 10464 fs_manager.cc:359] Metadata directory not provided
I20260925 10:44:01.998224 10464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:44:01.998567 10464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:44:02.001253 10464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/ts-1/data/instance:
uuid: "d94baa90bd21454a9c4d5cfa78d42a4f"
format_stamp: "Formatted at 2026-09-25 10:44:01 on dist-test-slave-dxdd"
server_key: "37b157b5a05da16b6da85ee301652277"
server_key_iv: "31c830ae31fdf5a03cd7898f025e3152"
server_key_version: "encryptionkey@0"
I20260925 10:44:02.002027 10464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/ts-1/wal/instance:
uuid: "d94baa90bd21454a9c4d5cfa78d42a4f"
format_stamp: "Formatted at 2026-09-25 10:44:01 on dist-test-slave-dxdd"
server_key: "37b157b5a05da16b6da85ee301652277"
server_key_iv: "31c830ae31fdf5a03cd7898f025e3152"
server_key_version: "encryptionkey@0"
I20260925 10:44:02.007081 10464 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260925 10:44:02.010222 10478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:44:02.011428 10464 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260925 10:44:02.011591 10464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/ts-1/wal
uuid: "d94baa90bd21454a9c4d5cfa78d42a4f"
format_stamp: "Formatted at 2026-09-25 10:44:01 on dist-test-slave-dxdd"
server_key: "37b157b5a05da16b6da85ee301652277"
server_key_iv: "31c830ae31fdf5a03cd7898f025e3152"
server_key_version: "encryptionkey@0"
I20260925 10:44:02.011824 10464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:44:02.032173 10464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:44:02.032925 10464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:44:02.033215 10464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:44:02.034094 10464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:44:02.035609 10464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:44:02.035740 10464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:44:02.035898 10464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:44:02.035960 10464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:44:02.067800 10464 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.2:41439
I20260925 10:44:02.067888 10590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.2:41439 every 8 connection(s)
I20260925 10:44:02.069041 10464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
I20260925 10:44:02.074679 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 10464
I20260925 10:44:02.074865 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/ts-1/wal/instance
I20260925 10:44:02.075423 23952 external_mini_cluster.cc:1428] Setting key 1d9b7d9f8a778b41478274c92b4f085d
I20260925 10:44:02.078449 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.3:0
--local_ip_for_outbound_sockets=127.23.100.3
--webserver_interface=127.23.100.3
--webserver_port=0
--tserver_master_addrs=127.23.100.62:43213
--builtin_ntp_servers=127.23.100.20:44801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 10:44:02.082633 10462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:02.071399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:45535 (local address 127.23.100.62:43213)
0925 10:44:02.071576 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:02.071587 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:02.072911 (+ 1324us) server_negotiation.cc:407] Connection header received
0925 10:44:02.074745 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:02.074769 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:02.074859 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:02.074986 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:02.076347 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:02.077406 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:02.079902 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:02.080290 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:02.080382 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:02.081903 (+ 1521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:02.081908 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:02.081911 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:44:02.081928 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:02.081958 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:44:02.081969 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:02.082037 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:02.082278 (+ 241us) server_negotiation.cc:299] Negotiation successful
0925 10:44:02.082417 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 10:44:02.084057 10593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:02.071913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:43213 (local address 127.23.100.2:45535)
0925 10:44:02.072750 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:02.072789 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 10:44:02.074444 (+ 1655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:44:02.075281 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:44:02.075294 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:44:02.075403 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:44:02.076159 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:02.076184 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:02.077572 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:02.077585 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:02.079753 (+ 2168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:02.079772 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:02.080504 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:02.080513 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:02.080845 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:02.081567 (+ 722us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:44:02.081572 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:44:02.081630 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:44:02.081729 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:44:02.082065 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:44:02.082078 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:44:02.082087 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 10:44:02.082263 (+ 176us) client_negotiation.cc:240] Negotiation successful
0925 10:44:02.082687 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":252,"threads_started":1}
I20260925 10:44:02.085507 10591 heartbeater.cc:347] Connected to a master server at 127.23.100.62:43213
I20260925 10:44:02.085826 10591 heartbeater.cc:464] Registering TS with master...
I20260925 10:44:02.086735 10591 heartbeater.cc:511] Master 127.23.100.62:43213 requested a full tablet report, sending...
I20260925 10:44:02.088099 10273 ts_manager.cc:194] Registered new tserver with Master: d94baa90bd21454a9c4d5cfa78d42a4f (127.23.100.2:41439)
I20260925 10:44:02.088959 10273 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.100.2:45535
W20260925 10:44:02.396463 10594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:44:02.396710 10594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:44:02.396797 10594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:44:02.403769 10594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:44:02.403865 10594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:44:02.403980 10594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.3
I20260925 10:44:02.412637 10594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:44801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.3
--webserver_port=0
--tserver_master_addrs=127.23.100.62:43213
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.3
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:44:02.413920 10594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:44:02.415764 10594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:44:02.428679 10599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:02.428812 10600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:02.428702 10602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:44:02.429387 10594 server_base.cc:1034] running on GCE node
I20260925 10:44:02.429878 10594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:44:02.430840 10594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:44:02.432035 10594 hybrid_clock.cc:648] HybridClock initialized: now 1790333042431994 us; error 40 us; skew 500 ppm
I20260925 10:44:02.432353 10594 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:44:02.434628 10594 webserver.cc:460] Webserver started at http://127.23.100.3:36685/ using document root <none> and password file <none>
I20260925 10:44:02.435313 10594 fs_manager.cc:359] Metadata directory not provided
I20260925 10:44:02.435427 10594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:44:02.435738 10594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:44:02.438681 10594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/ts-2/data/instance:
uuid: "10ee6b5bdeb345b88b4f6de0507886cb"
format_stamp: "Formatted at 2026-09-25 10:44:02 on dist-test-slave-dxdd"
server_key: "6ad7a412d855311409905319199aa06a"
server_key_iv: "025480d6c66c9decca7100947052425d"
server_key_version: "encryptionkey@0"
I20260925 10:44:02.439458 10594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/ts-2/wal/instance:
uuid: "10ee6b5bdeb345b88b4f6de0507886cb"
format_stamp: "Formatted at 2026-09-25 10:44:02 on dist-test-slave-dxdd"
server_key: "6ad7a412d855311409905319199aa06a"
server_key_iv: "025480d6c66c9decca7100947052425d"
server_key_version: "encryptionkey@0"
I20260925 10:44:02.444425 10594 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 10:44:02.447619 10608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:44:02.448820 10594 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 10:44:02.448985 10594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/ts-2/wal
uuid: "10ee6b5bdeb345b88b4f6de0507886cb"
format_stamp: "Formatted at 2026-09-25 10:44:02 on dist-test-slave-dxdd"
server_key: "6ad7a412d855311409905319199aa06a"
server_key_iv: "025480d6c66c9decca7100947052425d"
server_key_version: "encryptionkey@0"
I20260925 10:44:02.449191 10594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:44:02.464865 10594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:44:02.465596 10594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:44:02.465880 10594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:44:02.466835 10594 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:44:02.468180 10594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:44:02.468292 10594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:44:02.468429 10594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:44:02.468538 10594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:44:02.499804 10594 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.3:43119
I20260925 10:44:02.499886 10720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.3:43119 every 8 connection(s)
I20260925 10:44:02.501067 10594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
I20260925 10:44:02.504563 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 10594
I20260925 10:44:02.504740 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/ts-2/wal/instance
I20260925 10:44:02.505223 23952 external_mini_cluster.cc:1428] Setting key 40fd8e38f27f1b3e23ba793333b08a40
I20260925 10:44:02.513388 10462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:02.503565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:57033 (local address 127.23.100.62:43213)
0925 10:44:02.503763 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:02.503768 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:02.505135 (+ 1367us) server_negotiation.cc:407] Connection header received
0925 10:44:02.507178 (+ 2043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:02.507184 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:02.507272 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:02.507379 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:02.508472 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:02.509256 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:02.510864 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:02.511261 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:02.511396 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:02.512617 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:02.512621 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:02.512624 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:44:02.512642 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:02.512672 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:44:02.512683 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:02.512774 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:02.512989 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 10:44:02.513086 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 10:44:02.514269 10723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:02.504066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:43213 (local address 127.23.100.3:57033)
0925 10:44:02.504976 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:02.505013 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 10:44:02.506931 (+ 1918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:44:02.507647 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:44:02.507658 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:44:02.507776 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:44:02.508316 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:02.508331 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:02.509419 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:02.509423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:02.510717 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:02.510729 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:02.511475 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:02.511481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:02.511706 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:02.512302 (+ 596us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:44:02.512305 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:44:02.512347 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:44:02.512420 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:44:02.512773 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:44:02.512786 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:44:02.512795 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 10:44:02.512963 (+ 168us) client_negotiation.cc:240] Negotiation successful
0925 10:44:02.513315 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":293,"threads_started":1}
I20260925 10:44:02.515707 10721 heartbeater.cc:347] Connected to a master server at 127.23.100.62:43213
I20260925 10:44:02.516031 10721 heartbeater.cc:464] Registering TS with master...
I20260925 10:44:02.516741 10721 heartbeater.cc:511] Master 127.23.100.62:43213 requested a full tablet report, sending...
I20260925 10:44:02.517970 10273 ts_manager.cc:194] Registered new tserver with Master: 10ee6b5bdeb345b88b4f6de0507886cb (127.23.100.3:43119)
I20260925 10:44:02.519096 10273 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.100.3:57033
I20260925 10:44:02.520040 23952 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:44:02.529119 10462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:02.523799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55944 (local address 127.23.100.62:43213)
0925 10:44:02.524010 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:02.524017 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:02.524413 (+ 396us) server_negotiation.cc:407] Connection header received
0925 10:44:02.524533 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:02.524537 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:02.524594 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:02.524662 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:02.525088 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:02.525747 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:02.527074 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:02.527435 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:02.527526 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:02.528430 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:02.528434 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:02.528437 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:44:02.528455 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:02.528484 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:44:02.528495 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:02.528600 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:02.528737 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 10:44:02.528803 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 10:44:02.532509 10273 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55944:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:44:02.539436 10273 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:44:02.571831 10733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:02.550636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.2:41439 (local address 127.0.0.1:60798)
0925 10:44:02.563365 (+ 12729us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:02.563392 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 10:44:02.563545 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:44:02.564410 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:44:02.564420 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:44:02.564457 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:44:02.564924 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:02.564940 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:02.566823 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:02.566830 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:02.568999 (+ 2169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:02.569042 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:02.570933 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:02.570939 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:02.571149 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:02.571208 (+ 59us) client_negotiation.cc:769] Sending connection context
0925 10:44:02.571332 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 10:44:02.571507 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12553,"thread_start_us":11894,"threads_started":1}
I20260925 10:44:02.572322 10736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:02.551092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60798 (local address 127.23.100.2:41439)
0925 10:44:02.562638 (+ 11546us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:02.562651 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:02.563703 (+ 1052us) server_negotiation.cc:407] Connection header received
0925 10:44:02.563830 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:02.563842 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:02.564165 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:02.564388 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:44:02.565153 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:02.566701 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:02.569264 (+ 2563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:02.570738 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:02.570868 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:02.571122 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:02.571670 (+ 548us) server_negotiation.cc:299] Negotiation successful
0925 10:44:02.571841 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11314,"thread_start_us":11560,"threads_started":1}
I20260925 10:44:02.572647 10734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:02.550621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57228 (local address 127.23.100.1:40689)
0925 10:44:02.565017 (+ 14396us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:02.565028 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:02.565057 (+ 29us) server_negotiation.cc:407] Connection header received
0925 10:44:02.565180 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:02.565186 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:02.565395 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:02.565580 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:44:02.566496 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:02.567835 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:02.569902 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:02.571161 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:02.571324 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:02.571541 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:02.571920 (+ 379us) server_negotiation.cc:299] Negotiation successful
0925 10:44:02.572350 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14251,"thread_start_us":14160,"threads_started":1}
I20260925 10:44:02.572808 10735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:02.550404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:40689 (local address 127.0.0.1:57228)
0925 10:44:02.562919 (+ 12515us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:02.562953 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 10:44:02.563123 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:44:02.565810 (+ 2687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:44:02.565819 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:44:02.565844 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:44:02.566285 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:02.566297 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:02.567949 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:02.567956 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:02.569664 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:02.569677 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:02.571360 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:02.571372 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:02.571541 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:02.571580 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 10:44:02.571656 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 10:44:02.571782 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12268,"thread_start_us":12897,"threads_started":1}
I20260925 10:44:02.573938 10737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:02.552889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.3:43119 (local address 127.0.0.1:58862)
0925 10:44:02.565830 (+ 12941us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:02.565845 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:44:02.565959 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:44:02.566922 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:44:02.566930 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:44:02.566956 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:44:02.567370 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:02.567381 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:02.569219 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:02.569227 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:02.571309 (+ 2082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:02.571323 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:02.573231 (+ 1908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:02.573239 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:02.573413 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:02.573442 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 10:44:02.573513 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 10:44:02.573644 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12824,"thread_start_us":12691,"threads_started":1}
I20260925 10:44:02.574661 10738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:02.553516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58862 (local address 127.23.100.3:43119)
0925 10:44:02.565435 (+ 11919us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:02.565444 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:02.566094 (+ 650us) server_negotiation.cc:407] Connection header received
0925 10:44:02.566194 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:02.566203 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:02.566474 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:02.566657 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:44:02.567574 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:02.569051 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:02.571519 (+ 2468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:02.572847 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:02.573867 (+ 1020us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:02.574145 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:02.574216 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 10:44:02.574394 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11762,"thread_start_us":11625,"threads_started":1}
I20260925 10:44:02.576757 10526 tablet_service.cc:1467] Processing CreateTablet for tablet 3c17536bad334af1af0942cfb210cec2 (DEFAULT_TABLE table=test-table [id=c0af30960b9740e881cffb6d7b4fd21a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:44:02.576833 10395 tablet_service.cc:1467] Processing CreateTablet for tablet 3c17536bad334af1af0942cfb210cec2 (DEFAULT_TABLE table=test-table [id=c0af30960b9740e881cffb6d7b4fd21a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:44:02.577957 10526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c17536bad334af1af0942cfb210cec2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:44:02.578142 10395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c17536bad334af1af0942cfb210cec2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:44:02.577670 10656 tablet_service.cc:1467] Processing CreateTablet for tablet 3c17536bad334af1af0942cfb210cec2 (DEFAULT_TABLE table=test-table [id=c0af30960b9740e881cffb6d7b4fd21a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:44:02.579007 10656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c17536bad334af1af0942cfb210cec2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:44:02.589241 10739 tablet_bootstrap.cc:492] T 3c17536bad334af1af0942cfb210cec2 P d94baa90bd21454a9c4d5cfa78d42a4f: Bootstrap starting.
I20260925 10:44:02.592845 10739 tablet_bootstrap.cc:654] T 3c17536bad334af1af0942cfb210cec2 P d94baa90bd21454a9c4d5cfa78d42a4f: Neither blocks nor log segments found. Creating new log.
I20260925 10:44:02.592830 10740 tablet_bootstrap.cc:492] T 3c17536bad334af1af0942cfb210cec2 P ae8a12c32675488482d3eb3c171ede2d: Bootstrap starting.
I20260925 10:44:02.594197 10739 log.cc:824] T 3c17536bad334af1af0942cfb210cec2 P d94baa90bd21454a9c4d5cfa78d42a4f: Log is configured to *not* fsync() on all Append() calls
I20260925 10:44:02.594502 10741 tablet_bootstrap.cc:492] T 3c17536bad334af1af0942cfb210cec2 P 10ee6b5bdeb345b88b4f6de0507886cb: Bootstrap starting.
I20260925 10:44:02.596295 10740 tablet_bootstrap.cc:654] T 3c17536bad334af1af0942cfb210cec2 P ae8a12c32675488482d3eb3c171ede2d: Neither blocks nor log segments found. Creating new log.
I20260925 10:44:02.597502 10741 tablet_bootstrap.cc:654] T 3c17536bad334af1af0942cfb210cec2 P 10ee6b5bdeb345b88b4f6de0507886cb: Neither blocks nor log segments found. Creating new log.
I20260925 10:44:02.597481 10740 log.cc:824] T 3c17536bad334af1af0942cfb210cec2 P ae8a12c32675488482d3eb3c171ede2d: Log is configured to *not* fsync() on all Append() calls
I20260925 10:44:02.597853 10739 tablet_bootstrap.cc:492] T 3c17536bad334af1af0942cfb210cec2 P d94baa90bd21454a9c4d5cfa78d42a4f: No bootstrap required, opened a new log
I20260925 10:44:02.598186 10739 ts_tablet_manager.cc:1397] T 3c17536bad334af1af0942cfb210cec2 P d94baa90bd21454a9c4d5cfa78d42a4f: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20260925 10:44:02.598569 10741 log.cc:824] T 3c17536bad334af1af0942cfb210cec2 P 10ee6b5bdeb345b88b4f6de0507886cb: Log is configured to *not* fsync() on all Append() calls
I20260925 10:44:02.601500 10741 tablet_bootstrap.cc:492] T 3c17536bad334af1af0942cfb210cec2 P 10ee6b5bdeb345b88b4f6de0507886cb: No bootstrap required, opened a new log
I20260925 10:44:02.601565 10740 tablet_bootstrap.cc:492] T 3c17536bad334af1af0942cfb210cec2 P ae8a12c32675488482d3eb3c171ede2d: No bootstrap required, opened a new log
I20260925 10:44:02.601752 10741 ts_tablet_manager.cc:1397] T 3c17536bad334af1af0942cfb210cec2 P 10ee6b5bdeb345b88b4f6de0507886cb: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20260925 10:44:02.601801 10740 ts_tablet_manager.cc:1397] T 3c17536bad334af1af0942cfb210cec2 P ae8a12c32675488482d3eb3c171ede2d: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260925 10:44:02.608548 10740 raft_consensus.cc:348] T 3c17536bad334af1af0942cfb210cec2 P ae8a12c32675488482d3eb3c171ede2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8a12c32675488482d3eb3c171ede2d" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 40689 } } peers { permanent_uuid: "d94baa90bd21454a9c4d5cfa78d42a4f" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 41439 } } peers { permanent_uuid: "10ee6b5bdeb345b88b4f6de0507886cb" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 43119 } }
I20260925 10:44:02.608917 10740 raft_consensus.cc:374] T 3c17536bad334af1af0942cfb210cec2 P ae8a12c32675488482d3eb3c171ede2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:44:02.609053 10740 raft_consensus.cc:729] T 3c17536bad334af1af0942cfb210cec2 P ae8a12c32675488482d3eb3c171ede2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae8a12c32675488482d3eb3c171ede2d, State: Initialized, Role: FOLLOWER
I20260925 10:44:02.609038 10739 raft_consensus.cc:348] T 3c17536bad334af1af0942cfb210cec2 P d94baa90bd21454a9c4d5cfa78d42a4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8a12c32675488482d3eb3c171ede2d" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 40689 } } peers { permanent_uuid: "d94baa90bd21454a9c4d5cfa78d42a4f" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 41439 } } peers { permanent_uuid: "10ee6b5bdeb345b88b4f6de0507886cb" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 43119 } }
I20260925 10:44:02.609462 10739 raft_consensus.cc:374] T 3c17536bad334af1af0942cfb210cec2 P d94baa90bd21454a9c4d5cfa78d42a4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:44:02.609560 10739 raft_consensus.cc:729] T 3c17536bad334af1af0942cfb210cec2 P d94baa90bd21454a9c4d5cfa78d42a4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d94baa90bd21454a9c4d5cfa78d42a4f, State: Initialized, Role: FOLLOWER
I20260925 10:44:02.609730 10740 consensus_queue.cc:260] T 3c17536bad334af1af0942cfb210cec2 P ae8a12c32675488482d3eb3c171ede2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8a12c32675488482d3eb3c171ede2d" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 40689 } } peers { permanent_uuid: "d94baa90bd21454a9c4d5cfa78d42a4f" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 41439 } } peers { permanent_uuid: "10ee6b5bdeb345b88b4f6de0507886cb" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 43119 } }
I20260925 10:44:02.610112 10739 consensus_queue.cc:260] T 3c17536bad334af1af0942cfb210cec2 P d94baa90bd21454a9c4d5cfa78d42a4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8a12c32675488482d3eb3c171ede2d" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 40689 } } peers { permanent_uuid: "d94baa90bd21454a9c4d5cfa78d42a4f" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 41439 } } peers { permanent_uuid: "10ee6b5bdeb345b88b4f6de0507886cb" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 43119 } }
I20260925 10:44:02.610945 10460 heartbeater.cc:503] Master 127.23.100.62:43213 was elected leader, sending a full tablet report...
I20260925 10:44:02.611608 10740 ts_tablet_manager.cc:1428] T 3c17536bad334af1af0942cfb210cec2 P ae8a12c32675488482d3eb3c171ede2d: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260925 10:44:02.611917 10741 raft_consensus.cc:348] T 3c17536bad334af1af0942cfb210cec2 P 10ee6b5bdeb345b88b4f6de0507886cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8a12c32675488482d3eb3c171ede2d" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 40689 } } peers { permanent_uuid: "d94baa90bd21454a9c4d5cfa78d42a4f" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 41439 } } peers { permanent_uuid: "10ee6b5bdeb345b88b4f6de0507886cb" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 43119 } }
I20260925 10:44:02.612317 10741 raft_consensus.cc:374] T 3c17536bad334af1af0942cfb210cec2 P 10ee6b5bdeb345b88b4f6de0507886cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:44:02.612418 10741 raft_consensus.cc:729] T 3c17536bad334af1af0942cfb210cec2 P 10ee6b5bdeb345b88b4f6de0507886cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ee6b5bdeb345b88b4f6de0507886cb, State: Initialized, Role: FOLLOWER
I20260925 10:44:02.613094 10741 consensus_queue.cc:260] T 3c17536bad334af1af0942cfb210cec2 P 10ee6b5bdeb345b88b4f6de0507886cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8a12c32675488482d3eb3c171ede2d" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 40689 } } peers { permanent_uuid: "d94baa90bd21454a9c4d5cfa78d42a4f" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 41439 } } peers { permanent_uuid: "10ee6b5bdeb345b88b4f6de0507886cb" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 43119 } }
I20260925 10:44:02.614820 10721 heartbeater.cc:503] Master 127.23.100.62:43213 was elected leader, sending a full tablet report...
I20260925 10:44:02.615006 10741 ts_tablet_manager.cc:1428] T 3c17536bad334af1af0942cfb210cec2 P 10ee6b5bdeb345b88b4f6de0507886cb: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260925 10:44:02.615460 10591 heartbeater.cc:503] Master 127.23.100.62:43213 was elected leader, sending a full tablet report...
I20260925 10:44:02.615549 10739 ts_tablet_manager.cc:1428] T 3c17536bad334af1af0942cfb210cec2 P d94baa90bd21454a9c4d5cfa78d42a4f: Time spent starting tablet: real 0.017s user 0.012s sys 0.003s
W20260925 10:44:02.635435 10461 tablet.cc:2369] T 3c17536bad334af1af0942cfb210cec2 P ae8a12c32675488482d3eb3c171ede2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:44:02.752761 10722 tablet.cc:2369] T 3c17536bad334af1af0942cfb210cec2 P 10ee6b5bdeb345b88b4f6de0507886cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:44:02.821056 10592 tablet.cc:2369] T 3c17536bad334af1af0942cfb210cec2 P d94baa90bd21454a9c4d5cfa78d42a4f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:44:02.846416 10745 raft_consensus.cc:482] T 3c17536bad334af1af0942cfb210cec2 P ae8a12c32675488482d3eb3c171ede2d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:44:02.846776 10745 raft_consensus.cc:504] T 3c17536bad334af1af0942cfb210cec2 P ae8a12c32675488482d3eb3c171ede2d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8a12c32675488482d3eb3c171ede2d" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 40689 } } peers { permanent_uuid: "d94baa90bd21454a9c4d5cfa78d42a4f" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 41439 } } peers { permanent_uuid: "10ee6b5bdeb345b88b4f6de0507886cb" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 43119 } }
I20260925 10:44:02.848809 10745 leader_election.cc:288] T 3c17536bad334af1af0942cfb210cec2 P ae8a12c32675488482d3eb3c171ede2d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d94baa90bd21454a9c4d5cfa78d42a4f (127.23.100.2:41439), 10ee6b5bdeb345b88b4f6de0507886cb (127.23.100.3:43119)
I20260925 10:44:02.857679 10736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:02.849091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:33995 (local address 127.23.100.2:41439)
0925 10:44:02.849293 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:02.849299 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:02.850096 (+ 797us) server_negotiation.cc:407] Connection header received
0925 10:44:02.850404 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:02.850408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:02.850468 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:02.850544 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:44:02.851213 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:02.852259 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:02.855262 (+ 3003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:02.856455 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:02.856623 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:02.856699 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:02.857309 (+ 610us) server_negotiation.cc:299] Negotiation successful
0925 10:44:02.857429 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 10:44:02.857666 10738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:02.849572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:43717 (local address 127.23.100.3:43119)
0925 10:44:02.849809 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:02.849818 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:02.850437 (+ 619us) server_negotiation.cc:407] Connection header received
0925 10:44:02.850686 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:02.850692 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:02.850760 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:02.850829 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:44:02.851562 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:02.852534 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:02.855286 (+ 2752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:02.856496 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:02.856636 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:02.856699 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:02.857309 (+ 610us) server_negotiation.cc:299] Negotiation successful
0925 10:44:02.857429 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 10:44:02.858016 10748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:02.848979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.2:41439 (local address 127.23.100.1:33995)
0925 10:44:02.849930 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:02.849967 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 10:44:02.850184 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:44:02.850588 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:44:02.850597 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:44:02.850628 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:44:02.851038 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:02.851052 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:02.852402 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:02.852406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:02.855021 (+ 2615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:02.855043 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:02.856750 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:02.856757 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:02.857053 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:02.857147 (+ 94us) client_negotiation.cc:769] Sending connection context
0925 10:44:02.857258 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 10:44:02.857505 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":779,"mutex_wait_us":139,"thread_start_us":584,"threads_started":1}
I20260925 10:44:02.858018 10749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:02.849285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.3:43119 (local address 127.23.100.1:43717)
0925 10:44:02.850298 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:02.850341 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 10:44:02.850500 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:44:02.850893 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:44:02.850897 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:44:02.850933 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:44:02.851403 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:02.851412 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:02.852676 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:02.852682 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:02.855107 (+ 2425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:02.855118 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:02.856728 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:02.856737 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:02.856986 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:02.857144 (+ 158us) client_negotiation.cc:769] Sending connection context
0925 10:44:02.857253 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 10:44:02.857505 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":867,"mutex_wait_us":168,"thread_start_us":638,"threads_started":1}
I20260925 10:44:02.859370 10676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c17536bad334af1af0942cfb210cec2" candidate_uuid: "ae8a12c32675488482d3eb3c171ede2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee6b5bdeb345b88b4f6de0507886cb" is_pre_election: true
I20260925 10:44:02.859367 10546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c17536bad334af1af0942cfb210cec2" candidate_uuid: "ae8a12c32675488482d3eb3c171ede2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d94baa90bd21454a9c4d5cfa78d42a4f" is_pre_election: true
I20260925 10:44:02.859951 10546 raft_consensus.cc:2454] T 3c17536bad334af1af0942cfb210cec2 P d94baa90bd21454a9c4d5cfa78d42a4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae8a12c32675488482d3eb3c171ede2d in term 0.
I20260925 10:44:02.859951 10676 raft_consensus.cc:2454] T 3c17536bad334af1af0942cfb210cec2 P 10ee6b5bdeb345b88b4f6de0507886cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae8a12c32675488482d3eb3c171ede2d in term 0.
I20260925 10:44:02.860656 10351 leader_election.cc:302] T 3c17536bad334af1af0942cfb210cec2 P ae8a12c32675488482d3eb3c171ede2d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae8a12c32675488482d3eb3c171ede2d, d94baa90bd21454a9c4d5cfa78d42a4f; no voters:
I20260925 10:44:02.861114 10745 raft_consensus.cc:2781] T 3c17536bad334af1af0942cfb210cec2 P ae8a12c32675488482d3eb3c171ede2d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:44:02.861284 10745 raft_consensus.cc:482] T 3c17536bad334af1af0942cfb210cec2 P ae8a12c32675488482d3eb3c171ede2d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:44:02.861410 10745 raft_consensus.cc:3037] T 3c17536bad334af1af0942cfb210cec2 P ae8a12c32675488482d3eb3c171ede2d [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:44:02.863615 10745 raft_consensus.cc:504] T 3c17536bad334af1af0942cfb210cec2 P ae8a12c32675488482d3eb3c171ede2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8a12c32675488482d3eb3c171ede2d" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 40689 } } peers { permanent_uuid: "d94baa90bd21454a9c4d5cfa78d42a4f" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 41439 } } peers { permanent_uuid: "10ee6b5bdeb345b88b4f6de0507886cb" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 43119 } }
I20260925 10:44:02.864163 10745 leader_election.cc:288] T 3c17536bad334af1af0942cfb210cec2 P ae8a12c32675488482d3eb3c171ede2d [CANDIDATE]: Term 1 election: Requested vote from peers d94baa90bd21454a9c4d5cfa78d42a4f (127.23.100.2:41439), 10ee6b5bdeb345b88b4f6de0507886cb (127.23.100.3:43119)
I20260925 10:44:02.864655 10546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c17536bad334af1af0942cfb210cec2" candidate_uuid: "ae8a12c32675488482d3eb3c171ede2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d94baa90bd21454a9c4d5cfa78d42a4f"
I20260925 10:44:02.864648 10676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c17536bad334af1af0942cfb210cec2" candidate_uuid: "ae8a12c32675488482d3eb3c171ede2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ee6b5bdeb345b88b4f6de0507886cb"
I20260925 10:44:02.864851 10676 raft_consensus.cc:3037] T 3c17536bad334af1af0942cfb210cec2 P 10ee6b5bdeb345b88b4f6de0507886cb [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:44:02.864851 10546 raft_consensus.cc:3037] T 3c17536bad334af1af0942cfb210cec2 P d94baa90bd21454a9c4d5cfa78d42a4f [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:44:02.868050 10546 raft_consensus.cc:2454] T 3c17536bad334af1af0942cfb210cec2 P d94baa90bd21454a9c4d5cfa78d42a4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae8a12c32675488482d3eb3c171ede2d in term 1.
I20260925 10:44:02.868034 10676 raft_consensus.cc:2454] T 3c17536bad334af1af0942cfb210cec2 P 10ee6b5bdeb345b88b4f6de0507886cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae8a12c32675488482d3eb3c171ede2d in term 1.
I20260925 10:44:02.868506 10351 leader_election.cc:302] T 3c17536bad334af1af0942cfb210cec2 P ae8a12c32675488482d3eb3c171ede2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 10ee6b5bdeb345b88b4f6de0507886cb, ae8a12c32675488482d3eb3c171ede2d, d94baa90bd21454a9c4d5cfa78d42a4f; no voters:
I20260925 10:44:02.868829 10745 raft_consensus.cc:2781] T 3c17536bad334af1af0942cfb210cec2 P ae8a12c32675488482d3eb3c171ede2d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:44:02.869249 10745 raft_consensus.cc:686] T 3c17536bad334af1af0942cfb210cec2 P ae8a12c32675488482d3eb3c171ede2d [term 1 LEADER]: Becoming Leader. State: Replica: ae8a12c32675488482d3eb3c171ede2d, State: Running, Role: LEADER
I20260925 10:44:02.869733 10745 consensus_queue.cc:237] T 3c17536bad334af1af0942cfb210cec2 P ae8a12c32675488482d3eb3c171ede2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8a12c32675488482d3eb3c171ede2d" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 40689 } } peers { permanent_uuid: "d94baa90bd21454a9c4d5cfa78d42a4f" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 41439 } } peers { permanent_uuid: "10ee6b5bdeb345b88b4f6de0507886cb" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 43119 } }
I20260925 10:44:02.873972 10272 catalog_manager.cc:5495] T 3c17536bad334af1af0942cfb210cec2 P ae8a12c32675488482d3eb3c171ede2d reported cstate change: term changed from 0 to 1, leader changed from <none> to ae8a12c32675488482d3eb3c171ede2d (127.23.100.1). New cstate: current_term: 1 leader_uuid: "ae8a12c32675488482d3eb3c171ede2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8a12c32675488482d3eb3c171ede2d" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 40689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d94baa90bd21454a9c4d5cfa78d42a4f" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 41439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ee6b5bdeb345b88b4f6de0507886cb" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 43119 } health_report { overall_health: UNKNOWN } } }
I20260925 10:44:02.936246 10734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:02.931380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57244 (local address 127.23.100.1:40689)
0925 10:44:02.931570 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:02.931574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:02.931702 (+ 128us) server_negotiation.cc:407] Connection header received
0925 10:44:02.931865 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:02.931869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:02.931955 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:02.932076 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:44:02.932403 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:02.933044 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:02.934308 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:02.934688 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:02.934773 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:02.935246 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:44:02.935705 (+ 459us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:44:02.935826 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:02.935914 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 10:44:02.935946 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 10:44:02.943967 10546 raft_consensus.cc:1261] T 3c17536bad334af1af0942cfb210cec2 P d94baa90bd21454a9c4d5cfa78d42a4f [term 1 FOLLOWER]: Refusing update from remote peer ae8a12c32675488482d3eb3c171ede2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:44:02.944027 10676 raft_consensus.cc:1261] T 3c17536bad334af1af0942cfb210cec2 P 10ee6b5bdeb345b88b4f6de0507886cb [term 1 FOLLOWER]: Refusing update from remote peer ae8a12c32675488482d3eb3c171ede2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:44:02.944969 10750 consensus_queue.cc:1035] T 3c17536bad334af1af0942cfb210cec2 P ae8a12c32675488482d3eb3c171ede2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ee6b5bdeb345b88b4f6de0507886cb" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 43119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:44:02.945335 10745 consensus_queue.cc:1035] T 3c17536bad334af1af0942cfb210cec2 P ae8a12c32675488482d3eb3c171ede2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d94baa90bd21454a9c4d5cfa78d42a4f" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 41439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:44:02.957466 10754 mvcc.cc:205] Tried to move back new op lower bound from 7333204143889129472 to 7333204143598149632. Current Snapshot: MvccSnapshot[applied={T|T < 7333204143889129472}]
I20260925 10:44:02.966516 10273 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:44:02.966814 10273 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:44:02.971061 10273 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f8fb0dfd4256457689a9e86a84ed4a5c: Sending DeleteTablet for 3 replicas of tablet 3c17536bad334af1af0942cfb210cec2
I20260925 10:44:02.972278 10526 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c17536bad334af1af0942cfb210cec2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:44:02 UTC) from {username='slave'} at 127.0.0.1:60798
I20260925 10:44:02.972258 10525 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c17536bad334af1af0942cfb210cec2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:44:02 UTC) from {username='slave'} at 127.0.0.1:60798
I20260925 10:44:02.972561 10395 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c17536bad334af1af0942cfb210cec2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:44:02 UTC) from {username='slave'} at 127.0.0.1:57228
I20260925 10:44:02.973522 10764 tablet_replica.cc:330] T 3c17536bad334af1af0942cfb210cec2 P ae8a12c32675488482d3eb3c171ede2d: stopping tablet replica
I20260925 10:44:02.973848 10656 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c17536bad334af1af0942cfb210cec2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:44:02 UTC) from {username='slave'} at 127.0.0.1:58862
I20260925 10:44:02.974083 10764 raft_consensus.cc:2229] T 3c17536bad334af1af0942cfb210cec2 P ae8a12c32675488482d3eb3c171ede2d [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:44:02.974812 10764 raft_consensus.cc:2258] T 3c17536bad334af1af0942cfb210cec2 P ae8a12c32675488482d3eb3c171ede2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:44:02.975271 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 10333
I20260925 10:44:02.977104 10764 ts_tablet_manager.cc:1905] T 3c17536bad334af1af0942cfb210cec2 P ae8a12c32675488482d3eb3c171ede2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:44:02.986939 10764 ts_tablet_manager.cc:1918] T 3c17536bad334af1af0942cfb210cec2 P ae8a12c32675488482d3eb3c171ede2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:44:02.987175 10764 log.cc:1196] T 3c17536bad334af1af0942cfb210cec2 P ae8a12c32675488482d3eb3c171ede2d: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/ts-0/wal/wals/3c17536bad334af1af0942cfb210cec2
I20260925 10:44:02.987903 10764 ts_tablet_manager.cc:1939] T 3c17536bad334af1af0942cfb210cec2 P ae8a12c32675488482d3eb3c171ede2d: Deleting consensus metadata
I20260925 10:44:02.990021 10257 catalog_manager.cc:4841] TS ae8a12c32675488482d3eb3c171ede2d (127.23.100.1:40689): tablet 3c17536bad334af1af0942cfb210cec2 (table test-table [id=c0af30960b9740e881cffb6d7b4fd21a]) successfully deleted
I20260925 10:44:02.992908 10765 tablet_replica.cc:330] T 3c17536bad334af1af0942cfb210cec2 P 10ee6b5bdeb345b88b4f6de0507886cb: stopping tablet replica
I20260925 10:44:02.993472 10765 raft_consensus.cc:2229] T 3c17536bad334af1af0942cfb210cec2 P 10ee6b5bdeb345b88b4f6de0507886cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:44:02.994094 10765 raft_consensus.cc:2258] T 3c17536bad334af1af0942cfb210cec2 P 10ee6b5bdeb345b88b4f6de0507886cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:44:02.996429 10765 ts_tablet_manager.cc:1905] T 3c17536bad334af1af0942cfb210cec2 P 10ee6b5bdeb345b88b4f6de0507886cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:44:02.996994 10734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:02.976643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57250 (local address 127.23.100.1:40689)
0925 10:44:02.976806 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:02.976814 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:02.986251 (+ 9437us) server_negotiation.cc:407] Connection header received
0925 10:44:02.986620 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:02.986628 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:02.986759 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:02.986886 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:02.988197 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:02.989719 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:02.992100 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:02.993026 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:02.993177 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:02.994977 (+ 1800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:02.994991 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:02.995014 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:44:02.995984 (+ 970us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:02.996061 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:44:02.996090 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:02.996190 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:02.996491 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 10:44:02.996623 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 10:44:02.998032 10455 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:44:02.998446 10763 tablet_replica.cc:330] T 3c17536bad334af1af0942cfb210cec2 P d94baa90bd21454a9c4d5cfa78d42a4f: stopping tablet replica
I20260925 10:44:02.998932 10763 raft_consensus.cc:2229] T 3c17536bad334af1af0942cfb210cec2 P d94baa90bd21454a9c4d5cfa78d42a4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:44:02.999675 10763 raft_consensus.cc:2258] T 3c17536bad334af1af0942cfb210cec2 P d94baa90bd21454a9c4d5cfa78d42a4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:44:03.002596 10763 ts_tablet_manager.cc:1905] T 3c17536bad334af1af0942cfb210cec2 P d94baa90bd21454a9c4d5cfa78d42a4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:44:03.004271 10765 ts_tablet_manager.cc:1918] T 3c17536bad334af1af0942cfb210cec2 P 10ee6b5bdeb345b88b4f6de0507886cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:44:03.004420 10765 log.cc:1196] T 3c17536bad334af1af0942cfb210cec2 P 10ee6b5bdeb345b88b4f6de0507886cb: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/ts-2/wal/wals/3c17536bad334af1af0942cfb210cec2
I20260925 10:44:03.004885 10765 ts_tablet_manager.cc:1939] T 3c17536bad334af1af0942cfb210cec2 P 10ee6b5bdeb345b88b4f6de0507886cb: Deleting consensus metadata
I20260925 10:44:03.006633 10259 catalog_manager.cc:4841] TS 10ee6b5bdeb345b88b4f6de0507886cb (127.23.100.3:43119): tablet 3c17536bad334af1af0942cfb210cec2 (table test-table [id=c0af30960b9740e881cffb6d7b4fd21a]) successfully deleted
I20260925 10:44:03.008491 10763 ts_tablet_manager.cc:1918] T 3c17536bad334af1af0942cfb210cec2 P d94baa90bd21454a9c4d5cfa78d42a4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:44:03.008653 10763 log.cc:1196] T 3c17536bad334af1af0942cfb210cec2 P d94baa90bd21454a9c4d5cfa78d42a4f: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790332740062128-23952-0/minicluster-data/ts-1/wal/wals/3c17536bad334af1af0942cfb210cec2
I20260925 10:44:03.009119 10763 ts_tablet_manager.cc:1939] T 3c17536bad334af1af0942cfb210cec2 P d94baa90bd21454a9c4d5cfa78d42a4f: Deleting consensus metadata
I20260925 10:44:03.010517 10260 catalog_manager.cc:4841] TS d94baa90bd21454a9c4d5cfa78d42a4f (127.23.100.2:41439): tablet 3c17536bad334af1af0942cfb210cec2 (table test-table [id=c0af30960b9740e881cffb6d7b4fd21a]) successfully deleted
W20260925 10:44:03.011019 10260 catalog_manager.cc:4808] TS d94baa90bd21454a9c4d5cfa78d42a4f (127.23.100.2:41439): delete failed for tablet 3c17536bad334af1af0942cfb210cec2 because the tablet was not found. No further retry: Not found: Tablet not found: 3c17536bad334af1af0942cfb210cec2
I20260925 10:44:03.103993 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 10333
I20260925 10:44:03.121578 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 10464
I20260925 10:44:03.128540 10736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:03.122330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60812 (local address 127.23.100.2:41439)
0925 10:44:03.122522 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:03.122530 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:03.122623 (+ 93us) server_negotiation.cc:407] Connection header received
0925 10:44:03.122834 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:03.122840 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:03.122932 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:03.123041 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:03.123447 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:03.124149 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:03.125460 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:03.125844 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:03.125970 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:03.127214 (+ 1244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:03.127222 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:03.127236 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:44:03.127801 (+ 565us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:03.127846 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:44:03.127866 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:03.127988 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:03.128091 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 10:44:03.128206 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 10:44:03.129226 10586 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:44:03.215611 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 10464
I20260925 10:44:03.233361 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 10594
I20260925 10:44:03.240569 10738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:03.234135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58878 (local address 127.23.100.3:43119)
0925 10:44:03.234337 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:03.234345 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:03.234369 (+ 24us) server_negotiation.cc:407] Connection header received
0925 10:44:03.234590 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:03.234594 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:03.234737 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:03.234845 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:03.235255 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:03.236105 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:03.237497 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:03.237884 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:03.238013 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:03.239282 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:03.239291 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:03.239306 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:44:03.239862 (+ 556us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:03.239906 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:44:03.239930 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:03.240054 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:03.240155 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 10:44:03.240256 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 10:44:03.241182 10716 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:44:03.327876 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 10594
I20260925 10:44:03.344650 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 10242
I20260925 10:44:03.345706 10303 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:44:03.453971 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 10242
2026-09-25T10:44:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2866 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:44:03 dist-test-slave-dxdd krb5kdc[10773](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:44:03 dist-test-slave-dxdd krb5kdc[10773](info): set up 2 sockets
Sep 25 10:44:03 dist-test-slave-dxdd krb5kdc[10773](info): commencing operation
krb5kdc: starting...
W20260925 10:44:04.546584 23952 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Sep 25 10:44:04 dist-test-slave-dxdd krb5kdc[10773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:44:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:44:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
I20260925 10:44:04.781517 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:44381
--webserver_interface=127.23.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:34229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.62:44381
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:44:05.105350 10786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:44:05.105620 10786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:44:05.105787 10786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:44:05.112908 10786 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:44:05.113008 10786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:44:05.113052 10786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:44:05.113128 10786 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:44:05.113193 10786 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:44:05.123503 10786 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:34229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.62:44381
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:44381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:44:05.124770 10786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:44:05.126058 10786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:44:05.137543 10792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:05.137593 10791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:05.138255 10794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:44:05.138213 10786 server_base.cc:1034] running on GCE node
I20260925 10:44:05.139104 10786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:44:05.140407 10786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:44:05.141609 10786 hybrid_clock.cc:648] HybridClock initialized: now 1790333045141566 us; error 41 us; skew 500 ppm
I20260925 10:44:05.141947 10786 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:44:05 dist-test-slave-dxdd krb5kdc[10773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333045, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:44:05.144944 10786 init.cc:377] Logged in from keytab as kudu/127.23.100.62@KRBTEST.COM (short username kudu)
I20260925 10:44:05.146395 10786 webserver.cc:460] Webserver started at http://127.23.100.62:35677/ using document root <none> and password file <none>
I20260925 10:44:05.147006 10786 fs_manager.cc:359] Metadata directory not provided
I20260925 10:44:05.147104 10786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:44:05.147342 10786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:44:05.150141 10786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/master-0/data/instance:
uuid: "7b5fda6815ff4a139d4a1ef4e5be537b"
format_stamp: "Formatted at 2026-09-25 10:44:05 on dist-test-slave-dxdd"
server_key: "79da3e8506eadf2261a55770444a46a2"
server_key_iv: "de4d2e233cbf7fa6e85895b6cb824ca3"
server_key_version: "encryptionkey@0"
I20260925 10:44:05.150884 10786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/master-0/wal/instance:
uuid: "7b5fda6815ff4a139d4a1ef4e5be537b"
format_stamp: "Formatted at 2026-09-25 10:44:05 on dist-test-slave-dxdd"
server_key: "79da3e8506eadf2261a55770444a46a2"
server_key_iv: "de4d2e233cbf7fa6e85895b6cb824ca3"
server_key_version: "encryptionkey@0"
I20260925 10:44:05.155702 10786 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260925 10:44:05.159034 10801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:44:05.160267 10786 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 10:44:05.160441 10786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/master-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/master-0/wal
uuid: "7b5fda6815ff4a139d4a1ef4e5be537b"
format_stamp: "Formatted at 2026-09-25 10:44:05 on dist-test-slave-dxdd"
server_key: "79da3e8506eadf2261a55770444a46a2"
server_key_iv: "de4d2e233cbf7fa6e85895b6cb824ca3"
server_key_version: "encryptionkey@0"
I20260925 10:44:05.160655 10786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:44:05.180617 10786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:44:05.184522 10786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:44:05.184826 10786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:44:05.203415 10786 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.62:44381
I20260925 10:44:05.203437 10852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.62:44381 every 8 connection(s)
I20260925 10:44:05.204742 10786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
I20260925 10:44:05.207051 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 10786
I20260925 10:44:05.207257 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/master-0/wal/instance
I20260925 10:44:05.207686 23952 external_mini_cluster.cc:1428] Setting key 53f014af2cc0f5084b8f7d5a6e606c88
I20260925 10:44:05.208678 10853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 10:44:05 dist-test-slave-dxdd krb5kdc[10773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:44:05.222409 10853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b5fda6815ff4a139d4a1ef4e5be537b: Bootstrap starting.
I20260925 10:44:05.226455 10853 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b5fda6815ff4a139d4a1ef4e5be537b: Neither blocks nor log segments found. Creating new log.
I20260925 10:44:05.227710 10853 log.cc:824] T 00000000000000000000000000000000 P 7b5fda6815ff4a139d4a1ef4e5be537b: Log is configured to *not* fsync() on all Append() calls
I20260925 10:44:05.231050 10853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b5fda6815ff4a139d4a1ef4e5be537b: No bootstrap required, opened a new log
I20260925 10:44:05.232193 10856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:05.209552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58910 (local address 127.23.100.62:44381)
0925 10:44:05.210353 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:05.210367 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:05.210414 (+ 47us) server_negotiation.cc:407] Connection header received
0925 10:44:05.212144 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:05.212171 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:05.212648 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:05.213075 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:05.214442 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:05.216491 (+ 2049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:05.217805 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:05.218826 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:05.219075 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:05.223299 (+ 4224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:05.223308 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:05.223326 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:44:05.223370 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:05.227234 (+ 3864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:05.227961 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:05.227968 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:05.227973 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:05.228141 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:05.228549 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:05.228558 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:05.228569 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:05.229087 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:05.229387 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:05.230025 (+ 638us) server_negotiation.cc:299] Negotiation successful
0925 10:44:05.230758 (+ 733us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":306,"threads_started":1}
I20260925 10:44:05.239135 10853 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b5fda6815ff4a139d4a1ef4e5be537b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b5fda6815ff4a139d4a1ef4e5be537b" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 44381 } }
I20260925 10:44:05.239508 10853 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b5fda6815ff4a139d4a1ef4e5be537b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:44:05.239614 10853 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b5fda6815ff4a139d4a1ef4e5be537b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b5fda6815ff4a139d4a1ef4e5be537b, State: Initialized, Role: FOLLOWER
I20260925 10:44:05.240253 10853 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b5fda6815ff4a139d4a1ef4e5be537b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b5fda6815ff4a139d4a1ef4e5be537b" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 44381 } }
I20260925 10:44:05.240542 10853 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b5fda6815ff4a139d4a1ef4e5be537b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:44:05.240698 10853 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b5fda6815ff4a139d4a1ef4e5be537b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:44:05.240878 10853 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b5fda6815ff4a139d4a1ef4e5be537b [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:44:05.243049 10853 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b5fda6815ff4a139d4a1ef4e5be537b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b5fda6815ff4a139d4a1ef4e5be537b" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 44381 } }
I20260925 10:44:05.243520 10853 leader_election.cc:302] T 00000000000000000000000000000000 P 7b5fda6815ff4a139d4a1ef4e5be537b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b5fda6815ff4a139d4a1ef4e5be537b; no voters:
I20260925 10:44:05.244239 10853 leader_election.cc:288] T 00000000000000000000000000000000 P 7b5fda6815ff4a139d4a1ef4e5be537b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:44:05.244329 10858 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b5fda6815ff4a139d4a1ef4e5be537b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:44:05.244961 10858 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b5fda6815ff4a139d4a1ef4e5be537b [term 1 LEADER]: Becoming Leader. State: Replica: 7b5fda6815ff4a139d4a1ef4e5be537b, State: Running, Role: LEADER
I20260925 10:44:05.245571 10858 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b5fda6815ff4a139d4a1ef4e5be537b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b5fda6815ff4a139d4a1ef4e5be537b" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 44381 } }
I20260925 10:44:05.246472 10853 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b5fda6815ff4a139d4a1ef4e5be537b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:44:05.248431 10859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b5fda6815ff4a139d4a1ef4e5be537b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b5fda6815ff4a139d4a1ef4e5be537b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b5fda6815ff4a139d4a1ef4e5be537b" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 44381 } } }
I20260925 10:44:05.248626 10860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b5fda6815ff4a139d4a1ef4e5be537b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b5fda6815ff4a139d4a1ef4e5be537b. Latest consensus state: current_term: 1 leader_uuid: "7b5fda6815ff4a139d4a1ef4e5be537b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b5fda6815ff4a139d4a1ef4e5be537b" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 44381 } } }
I20260925 10:44:05.248874 10859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b5fda6815ff4a139d4a1ef4e5be537b [sys.catalog]: This master's current role is: LEADER
I20260925 10:44:05.248955 10860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b5fda6815ff4a139d4a1ef4e5be537b [sys.catalog]: This master's current role is: LEADER
I20260925 10:44:05.249889 10864 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:44:05.255857 10864 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:44:05.264097 10864 catalog_manager.cc:1347] Generated new cluster ID: deb3e18dd6c94bf68ef9bae009d0b6ea
I20260925 10:44:05.264218 10864 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:44:05.274452 10864 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:44:05.275627 10864 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:44:05.282099 10864 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b5fda6815ff4a139d4a1ef4e5be537b: Generated new TSK 0
I20260925 10:44:05.282895 10864 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab.
Entry for principal kudu/127.23.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab.
I20260925 10:44:05.349709 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.1:0
--local_ip_for_outbound_sockets=127.23.100.1
--webserver_interface=127.23.100.1
--webserver_port=0
--tserver_master_addrs=127.23.100.62:44381
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab
--principal=kudu/127.23.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:34229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:44:05.678404 10879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:44:05.678668 10879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:44:05.678866 10879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:44:05.685920 10879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:44:05.686012 10879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:44:05.686154 10879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.1
I20260925 10:44:05.695631 10879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:34229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.1.keytab
--principal=kudu/127.23.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.1
--webserver_port=0
--tserver_master_addrs=127.23.100.62:44381
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.1
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:44:05.696833 10879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:44:05.698087 10879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:44:05.710860 10884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:05.710894 10885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:05.710820 10887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:44:05.711931 10879 server_base.cc:1034] running on GCE node
I20260925 10:44:05.712503 10879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:44:05.713379 10879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:44:05.714608 10879 hybrid_clock.cc:648] HybridClock initialized: now 1790333045714550 us; error 50 us; skew 500 ppm
I20260925 10:44:05.714907 10879 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:44:05 dist-test-slave-dxdd krb5kdc[10773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333045, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:44:05.717770 10879 init.cc:377] Logged in from keytab as kudu/127.23.100.1@KRBTEST.COM (short username kudu)
I20260925 10:44:05.719144 10879 webserver.cc:460] Webserver started at http://127.23.100.1:41977/ using document root <none> and password file <none>
I20260925 10:44:05.719777 10879 fs_manager.cc:359] Metadata directory not provided
I20260925 10:44:05.719918 10879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:44:05.720225 10879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:44:05.723017 10879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/ts-0/data/instance:
uuid: "f321f99640ce40c69d7a5e0e48a34d8b"
format_stamp: "Formatted at 2026-09-25 10:44:05 on dist-test-slave-dxdd"
server_key: "224dea44b09ad542fe281d49b9160004"
server_key_iv: "75ae2035dc19405bdeb4876a35ddcc67"
server_key_version: "encryptionkey@0"
I20260925 10:44:05.723755 10879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/ts-0/wal/instance:
uuid: "f321f99640ce40c69d7a5e0e48a34d8b"
format_stamp: "Formatted at 2026-09-25 10:44:05 on dist-test-slave-dxdd"
server_key: "224dea44b09ad542fe281d49b9160004"
server_key_iv: "75ae2035dc19405bdeb4876a35ddcc67"
server_key_version: "encryptionkey@0"
I20260925 10:44:05.728295 10879 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 10:44:05.731757 10894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:44:05.733084 10879 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260925 10:44:05.733206 10879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/ts-0/wal
uuid: "f321f99640ce40c69d7a5e0e48a34d8b"
format_stamp: "Formatted at 2026-09-25 10:44:05 on dist-test-slave-dxdd"
server_key: "224dea44b09ad542fe281d49b9160004"
server_key_iv: "75ae2035dc19405bdeb4876a35ddcc67"
server_key_version: "encryptionkey@0"
I20260925 10:44:05.733350 10879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:44:05.747701 10879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:44:05.751420 10879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:44:05.751631 10879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:44:05.752475 10879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:44:05.753638 10879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:44:05.753706 10879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:44:05.753782 10879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:44:05.753829 10879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:44:05.786082 10879 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.1:45163
I20260925 10:44:05.786145 11006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.1:45163 every 8 connection(s)
I20260925 10:44:05.787439 10879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
I20260925 10:44:05.793550 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 10879
I20260925 10:44:05.793742 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/ts-0/wal/instance
I20260925 10:44:05.794340 23952 external_mini_cluster.cc:1428] Setting key 0867c06e9ab0ff68d4023763933c2a2e
Sep 25 10:44:05 dist-test-slave-dxdd krb5kdc[10773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333045, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.1@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:44:05.808810 11009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:05.789808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:51971 (local address 127.23.100.62:44381)
0925 10:44:05.790272 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:05.790278 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:05.791249 (+ 971us) server_negotiation.cc:407] Connection header received
0925 10:44:05.792901 (+ 1652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:05.792905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:05.792979 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:05.793074 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:05.795112 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:05.796135 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:05.798533 (+ 2398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:05.798909 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:05.799046 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:05.803201 (+ 4155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:05.803206 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:05.803209 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:44:05.803239 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:05.805746 (+ 2507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:05.806525 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:05.806530 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:05.806532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:05.806656 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:05.807114 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:05.807118 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:05.807120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:05.807364 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:05.807545 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:05.808202 (+ 657us) server_negotiation.cc:299] Negotiation successful
0925 10:44:05.808448 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":212,"threads_started":1}
I20260925 10:44:05.809695 11010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:05.790423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44381 (local address 127.23.100.1:51971)
0925 10:44:05.791099 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:05.791135 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 10:44:05.792668 (+ 1533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:44:05.793355 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:44:05.793368 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:44:05.794087 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:44:05.794930 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:05.794961 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:05.796320 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:05.796325 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:05.798399 (+ 2074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:05.798416 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:05.799182 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:05.799191 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:05.799521 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:05.800211 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:44:05.800262 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:44:05.802968 (+ 2706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:44:05.805886 (+ 2918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:44:05.805894 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:44:05.805909 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:44:05.806383 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:44:05.806769 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:44:05.806774 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:44:05.806778 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:44:05.806985 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:44:05.807513 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:44:05.807520 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:44:05.807840 (+ 320us) client_negotiation.cc:769] Sending connection context
0925 10:44:05.808091 (+ 251us) client_negotiation.cc:240] Negotiation successful
0925 10:44:05.808596 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":254,"threads_started":1}
I20260925 10:44:05.811391 11007 heartbeater.cc:347] Connected to a master server at 127.23.100.62:44381
I20260925 10:44:05.811700 11007 heartbeater.cc:464] Registering TS with master...
I20260925 10:44:05.812547 11007 heartbeater.cc:511] Master 127.23.100.62:44381 requested a full tablet report, sending...
I20260925 10:44:05.814850 10818 ts_manager.cc:194] Registered new tserver with Master: f321f99640ce40c69d7a5e0e48a34d8b (127.23.100.1:45163)
I20260925 10:44:05.816984 10818 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.1@KRBTEST.COM'} at 127.23.100.1:51971
WARNING: no policy specified for kudu/127.23.100.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab.
Entry for principal kudu/127.23.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab.
I20260925 10:44:05.860680 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.2:0
--local_ip_for_outbound_sockets=127.23.100.2
--webserver_interface=127.23.100.2
--webserver_port=0
--tserver_master_addrs=127.23.100.62:44381
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab
--principal=kudu/127.23.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:34229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:44:06.186875 11013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:44:06.187145 11013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:44:06.187302 11013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:44:06.194274 11013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:44:06.194453 11013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:44:06.194674 11013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.2
I20260925 10:44:06.204679 11013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:34229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.2.keytab
--principal=kudu/127.23.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.2
--webserver_port=0
--tserver_master_addrs=127.23.100.62:44381
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.2
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:44:06.205847 11013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:44:06.207152 11013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:44:06.221320 11018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:06.221416 11021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:06.221371 11019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:44:06.222251 11013 server_base.cc:1034] running on GCE node
I20260925 10:44:06.222901 11013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:44:06.223834 11013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:44:06.225041 11013 hybrid_clock.cc:648] HybridClock initialized: now 1790333046224990 us; error 25 us; skew 500 ppm
I20260925 10:44:06.225346 11013 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:44:06 dist-test-slave-dxdd krb5kdc[10773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333046, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:44:06.232726 11013 init.cc:377] Logged in from keytab as kudu/127.23.100.2@KRBTEST.COM (short username kudu)
I20260925 10:44:06.234599 11013 webserver.cc:460] Webserver started at http://127.23.100.2:36123/ using document root <none> and password file <none>
I20260925 10:44:06.235468 11013 fs_manager.cc:359] Metadata directory not provided
I20260925 10:44:06.235606 11013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:44:06.235965 11013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:44:06.239310 11013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/ts-1/data/instance:
uuid: "a5eb978f48154705846247c973ebb58c"
format_stamp: "Formatted at 2026-09-25 10:44:06 on dist-test-slave-dxdd"
server_key: "fdfbfeb195fb03bde22c1cec6a12edb2"
server_key_iv: "18a99b58771c4fd7a8e8470c5841c959"
server_key_version: "encryptionkey@0"
I20260925 10:44:06.240007 11013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/ts-1/wal/instance:
uuid: "a5eb978f48154705846247c973ebb58c"
format_stamp: "Formatted at 2026-09-25 10:44:06 on dist-test-slave-dxdd"
server_key: "fdfbfeb195fb03bde22c1cec6a12edb2"
server_key_iv: "18a99b58771c4fd7a8e8470c5841c959"
server_key_version: "encryptionkey@0"
I20260925 10:44:06.244890 11013 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 10:44:06.248205 11028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:44:06.249441 11013 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 10:44:06.249619 11013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/ts-1/wal
uuid: "a5eb978f48154705846247c973ebb58c"
format_stamp: "Formatted at 2026-09-25 10:44:06 on dist-test-slave-dxdd"
server_key: "fdfbfeb195fb03bde22c1cec6a12edb2"
server_key_iv: "18a99b58771c4fd7a8e8470c5841c959"
server_key_version: "encryptionkey@0"
I20260925 10:44:06.249805 11013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:44:06.276372 11013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:44:06.280320 11013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:44:06.280607 11013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:44:06.281569 11013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:44:06.282896 11013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:44:06.283010 11013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:44:06.283167 11013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:44:06.283241 11013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:44:06.315788 11013 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.2:39429
I20260925 10:44:06.315838 11140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.2:39429 every 8 connection(s)
I20260925 10:44:06.317090 11013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
I20260925 10:44:06.319306 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 11013
I20260925 10:44:06.319489 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/ts-1/wal/instance
I20260925 10:44:06.320061 23952 external_mini_cluster.cc:1428] Setting key d7d1d49bbfd12997c80636c64038c798
Sep 25 10:44:06 dist-test-slave-dxdd krb5kdc[10773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333046, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.2@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:44:06.339545 11143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:06.319814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:34269 (local address 127.23.100.62:44381)
0925 10:44:06.320371 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:06.320377 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:06.321331 (+ 954us) server_negotiation.cc:407] Connection header received
0925 10:44:06.323325 (+ 1994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:06.323330 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:06.323408 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:06.323537 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:06.325041 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:06.325794 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:06.327373 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:06.327832 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:06.327963 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:06.331839 (+ 3876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:06.331849 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:06.331857 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:44:06.331901 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:06.335853 (+ 3952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:06.336706 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:06.336714 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:06.336720 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:06.336875 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:06.337357 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:06.337365 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:06.337370 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:06.337794 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:06.337941 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:06.338793 (+ 852us) server_negotiation.cc:299] Negotiation successful
0925 10:44:06.339057 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":233,"threads_started":1}
I20260925 10:44:06.340147 11144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:06.320453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44381 (local address 127.23.100.2:34269)
0925 10:44:06.321185 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:06.321221 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 10:44:06.323061 (+ 1840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:44:06.323777 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:44:06.323785 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:44:06.324295 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:44:06.324871 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:06.324888 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:06.325949 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:06.325953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:06.327224 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:06.327239 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:06.328044 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:06.328051 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:06.328276 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:06.328969 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:44:06.328991 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:44:06.331592 (+ 2601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:44:06.336018 (+ 4426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:44:06.336025 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:44:06.336038 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:44:06.336572 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:44:06.337010 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:44:06.337013 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:44:06.337016 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:44:06.337204 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:44:06.337945 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:44:06.337951 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:44:06.338281 (+ 330us) client_negotiation.cc:769] Sending connection context
0925 10:44:06.338625 (+ 344us) client_negotiation.cc:240] Negotiation successful
0925 10:44:06.339042 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":315,"threads_started":1}
I20260925 10:44:06.341704 11141 heartbeater.cc:347] Connected to a master server at 127.23.100.62:44381
I20260925 10:44:06.342033 11141 heartbeater.cc:464] Registering TS with master...
I20260925 10:44:06.342903 11141 heartbeater.cc:511] Master 127.23.100.62:44381 requested a full tablet report, sending...
I20260925 10:44:06.344542 10818 ts_manager.cc:194] Registered new tserver with Master: a5eb978f48154705846247c973ebb58c (127.23.100.2:39429)
I20260925 10:44:06.345363 10818 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.2@KRBTEST.COM'} at 127.23.100.2:34269
WARNING: no policy specified for kudu/127.23.100.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab.
Entry for principal kudu/127.23.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab.
I20260925 10:44:06.379212 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.3:0
--local_ip_for_outbound_sockets=127.23.100.3
--webserver_interface=127.23.100.3
--webserver_port=0
--tserver_master_addrs=127.23.100.62:44381
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab
--principal=kudu/127.23.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:34229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:44:06.692513 11147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:44:06.692819 11147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:44:06.692984 11147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:44:06.700080 11147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:44:06.700227 11147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:44:06.700421 11147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.3
I20260925 10:44:06.710352 11147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:34229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.3.keytab
--principal=kudu/127.23.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.3
--webserver_port=0
--tserver_master_addrs=127.23.100.62:44381
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.3
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:44:06.711534 11147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:44:06.712741 11147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:44:06.725278 11155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:06.725399 11152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:06.725373 11153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:44:06.726094 11147 server_base.cc:1034] running on GCE node
I20260925 10:44:06.726663 11147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:44:06.727418 11147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:44:06.728648 11147 hybrid_clock.cc:648] HybridClock initialized: now 1790333046728629 us; error 66 us; skew 500 ppm
I20260925 10:44:06.728940 11147 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:44:06 dist-test-slave-dxdd krb5kdc[10773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333046, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:44:06.731846 11147 init.cc:377] Logged in from keytab as kudu/127.23.100.3@KRBTEST.COM (short username kudu)
I20260925 10:44:06.733206 11147 webserver.cc:460] Webserver started at http://127.23.100.3:35093/ using document root <none> and password file <none>
I20260925 10:44:06.733840 11147 fs_manager.cc:359] Metadata directory not provided
I20260925 10:44:06.733981 11147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:44:06.734285 11147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:44:06.737044 11147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/ts-2/data/instance:
uuid: "30253ab5c6e84ceda1456c89469563b4"
format_stamp: "Formatted at 2026-09-25 10:44:06 on dist-test-slave-dxdd"
server_key: "4e8e89c4b3ce9fe3f5e7db9e1153e711"
server_key_iv: "a00739f11583ef23a5bbfde4a127c850"
server_key_version: "encryptionkey@0"
I20260925 10:44:06.737756 11147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/ts-2/wal/instance:
uuid: "30253ab5c6e84ceda1456c89469563b4"
format_stamp: "Formatted at 2026-09-25 10:44:06 on dist-test-slave-dxdd"
server_key: "4e8e89c4b3ce9fe3f5e7db9e1153e711"
server_key_iv: "a00739f11583ef23a5bbfde4a127c850"
server_key_version: "encryptionkey@0"
I20260925 10:44:06.742636 11147 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260925 10:44:06.745733 11162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:44:06.746949 11147 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 10:44:06.747121 11147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/ts-2/wal
uuid: "30253ab5c6e84ceda1456c89469563b4"
format_stamp: "Formatted at 2026-09-25 10:44:06 on dist-test-slave-dxdd"
server_key: "4e8e89c4b3ce9fe3f5e7db9e1153e711"
server_key_iv: "a00739f11583ef23a5bbfde4a127c850"
server_key_version: "encryptionkey@0"
I20260925 10:44:06.747337 11147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:44:06.795469 11147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:44:06.799386 11147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:44:06.799692 11147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:44:06.800606 11147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:44:06.801891 11147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:44:06.801993 11147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:44:06.802141 11147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:44:06.802225 11147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:44:06.820906 11007 heartbeater.cc:503] Master 127.23.100.62:44381 was elected leader, sending a full tablet report...
I20260925 10:44:06.834703 11147 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.3:42039
I20260925 10:44:06.834784 11274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.3:42039 every 8 connection(s)
I20260925 10:44:06.835920 11147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
I20260925 10:44:06.842111 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 11147
I20260925 10:44:06.842296 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/ts-2/wal/instance
I20260925 10:44:06.842834 23952 external_mini_cluster.cc:1428] Setting key 64a4a3ee99e4b5c9dfcdf1b43b79cd3b
Sep 25 10:44:06 dist-test-slave-dxdd krb5kdc[10773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333046, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.3@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:44:06.855808 11143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:06.838558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:35061 (local address 127.23.100.62:44381)
0925 10:44:06.838730 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:06.838734 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:06.839930 (+ 1196us) server_negotiation.cc:407] Connection header received
0925 10:44:06.841915 (+ 1985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:06.841919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:06.841984 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:06.842067 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:06.844060 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:06.844956 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:06.846653 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:06.847086 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:06.847227 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:06.850532 (+ 3305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:06.850536 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:06.850539 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:44:06.850571 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:06.853214 (+ 2643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:06.853900 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:06.853903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:06.853905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:06.854015 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:06.854449 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:06.854453 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:06.854455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:06.854725 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:06.854839 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:06.855423 (+ 584us) server_negotiation.cc:299] Negotiation successful
0925 10:44:06.855577 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 10:44:06.856885 11277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:06.839121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44381 (local address 127.23.100.3:35061)
0925 10:44:06.839784 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:06.839821 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 10:44:06.841615 (+ 1794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:44:06.842381 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:44:06.842392 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:44:06.843098 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:44:06.843848 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:06.843868 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:06.845097 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:06.845103 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:06.846494 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:06.846505 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:06.847308 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:06.847314 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:06.847538 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:06.848243 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:44:06.848268 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:44:06.850283 (+ 2015us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:44:06.853369 (+ 3086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:44:06.853375 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:44:06.853386 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:44:06.853787 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:44:06.854119 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:44:06.854123 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:44:06.854125 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:44:06.854293 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:44:06.854826 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:44:06.854832 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:44:06.855120 (+ 288us) client_negotiation.cc:769] Sending connection context
0925 10:44:06.855329 (+ 209us) client_negotiation.cc:240] Negotiation successful
0925 10:44:06.855672 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":265,"threads_started":1}
I20260925 10:44:06.858275 11275 heartbeater.cc:347] Connected to a master server at 127.23.100.62:44381
I20260925 10:44:06.858603 11275 heartbeater.cc:464] Registering TS with master...
I20260925 10:44:06.859270 11275 heartbeater.cc:511] Master 127.23.100.62:44381 requested a full tablet report, sending...
I20260925 10:44:06.860502 10818 ts_manager.cc:194] Registered new tserver with Master: 30253ab5c6e84ceda1456c89469563b4 (127.23.100.3:42039)
I20260925 10:44:06.861222 10818 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.3@KRBTEST.COM'} at 127.23.100.3:35061
I20260925 10:44:06.869609 23952 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:44:06.885161 11143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:06.873577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58926 (local address 127.23.100.62:44381)
0925 10:44:06.873813 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:06.873822 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:06.874341 (+ 519us) server_negotiation.cc:407] Connection header received
0925 10:44:06.874481 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:06.874485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:06.874548 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:06.874626 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:06.875869 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:06.876544 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:06.877770 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:06.878160 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:06.878249 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:06.880717 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:06.880722 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:06.880725 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:44:06.880752 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:06.883118 (+ 2366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:06.883668 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:06.883672 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:06.883674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:06.883778 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:06.884070 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:06.884072 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:06.884074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:06.884346 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:06.884536 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:06.884778 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 10:44:06.884918 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 10:44:06.888516 10818 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58926:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:44:06.895243 10818 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:44:06.931367 11289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:06.906667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:45163 (local address 127.0.0.1:50790)
0925 10:44:06.913996 (+ 7329us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:06.914042 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 10:44:06.914233 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:44:06.923949 (+ 9716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:44:06.923958 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:44:06.923986 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:44:06.924419 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:06.924433 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:06.926593 (+ 2160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:06.926598 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:06.928394 (+ 1796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:06.928406 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:06.930275 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:06.930281 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:06.930505 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:06.930565 (+ 60us) client_negotiation.cc:769] Sending connection context
0925 10:44:06.930696 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 10:44:06.930889 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7029,"mutex_wait_us":64,"thread_start_us":16020,"threads_started":1}
I20260925 10:44:06.931727 11291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:06.907686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54440 (local address 127.23.100.3:42039)
0925 10:44:06.921829 (+ 14143us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:06.921840 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:06.923357 (+ 1517us) server_negotiation.cc:407] Connection header received
0925 10:44:06.923520 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:06.923526 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:06.923700 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:06.923881 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:44:06.924772 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:06.926236 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:06.929110 (+ 2874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:06.930570 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:06.930707 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:06.930917 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:06.931162 (+ 245us) server_negotiation.cc:299] Negotiation successful
0925 10:44:06.931335 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13931,"thread_start_us":14150,"threads_started":1}
I20260925 10:44:06.932118 11288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:06.907001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50790 (local address 127.23.100.1:45163)
0925 10:44:06.923229 (+ 16228us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:06.923236 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:06.923265 (+ 29us) server_negotiation.cc:407] Connection header received
0925 10:44:06.923382 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:06.923393 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:06.923675 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:06.923912 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:44:06.924657 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:06.926437 (+ 1780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:06.928553 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:06.930084 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:06.930231 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:06.930494 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:06.931607 (+ 1113us) server_negotiation.cc:299] Negotiation successful
0925 10:44:06.931782 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16051,"thread_start_us":15892,"threads_started":1}
I20260925 10:44:06.932272 11287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:06.907500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.2:39429 (local address 127.0.0.1:54908)
0925 10:44:06.923680 (+ 16180us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:06.923693 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:44:06.923820 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:44:06.924757 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:44:06.924761 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:44:06.924784 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:44:06.925128 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:06.925141 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:06.926887 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:06.926892 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:06.929131 (+ 2239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:06.929307 (+ 176us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:06.931153 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:06.931160 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:06.931347 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:06.931373 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 10:44:06.931711 (+ 338us) client_negotiation.cc:240] Negotiation successful
0925 10:44:06.931850 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16093,"mutex_wait_us":6,"thread_start_us":15451,"threads_started":1}
I20260925 10:44:06.932560 11292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:06.908125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54908 (local address 127.23.100.2:39429)
0925 10:44:06.922501 (+ 14376us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:06.922512 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:06.923801 (+ 1289us) server_negotiation.cc:407] Connection header received
0925 10:44:06.924021 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:06.924032 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:06.924267 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:06.924435 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:44:06.925306 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:06.926742 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:06.929464 (+ 2722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:06.930926 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:06.931102 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:06.931284 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:06.931772 (+ 488us) server_negotiation.cc:299] Negotiation successful
0925 10:44:06.932082 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14192,"thread_start_us":14037,"threads_started":1}
I20260925 10:44:06.932686 11290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:06.907081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.3:42039 (local address 127.0.0.1:54440)
0925 10:44:06.923227 (+ 16146us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:06.923246 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:44:06.923380 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:44:06.924185 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:44:06.924189 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:44:06.924209 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:44:06.924588 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:06.924600 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:06.926409 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:06.926416 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:06.928913 (+ 2497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:06.928927 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:06.930769 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:06.930775 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:06.930941 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:06.930971 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 10:44:06.931978 (+ 1007us) client_negotiation.cc:240] Negotiation successful
0925 10:44:06.932113 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15993,"mutex_wait_us":64,"thread_start_us":6463,"threads_started":1}
I20260925 10:44:06.936190 11210 tablet_service.cc:1467] Processing CreateTablet for tablet 032d30c0ca0647ab973dcdf8fd443221 (DEFAULT_TABLE table=test-table [id=fd0117ff8e98470192a0b3994aef480a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:44:06.936242 11076 tablet_service.cc:1467] Processing CreateTablet for tablet 032d30c0ca0647ab973dcdf8fd443221 (DEFAULT_TABLE table=test-table [id=fd0117ff8e98470192a0b3994aef480a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:44:06.936249 10942 tablet_service.cc:1467] Processing CreateTablet for tablet 032d30c0ca0647ab973dcdf8fd443221 (DEFAULT_TABLE table=test-table [id=fd0117ff8e98470192a0b3994aef480a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:44:06.937530 11210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 032d30c0ca0647ab973dcdf8fd443221. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:44:06.937553 11076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 032d30c0ca0647ab973dcdf8fd443221. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:44:06.937566 10942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 032d30c0ca0647ab973dcdf8fd443221. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:44:06.948431 11293 tablet_bootstrap.cc:492] T 032d30c0ca0647ab973dcdf8fd443221 P 30253ab5c6e84ceda1456c89469563b4: Bootstrap starting.
I20260925 10:44:06.951785 11293 tablet_bootstrap.cc:654] T 032d30c0ca0647ab973dcdf8fd443221 P 30253ab5c6e84ceda1456c89469563b4: Neither blocks nor log segments found. Creating new log.
I20260925 10:44:06.952960 11293 log.cc:824] T 032d30c0ca0647ab973dcdf8fd443221 P 30253ab5c6e84ceda1456c89469563b4: Log is configured to *not* fsync() on all Append() calls
I20260925 10:44:06.955940 11293 tablet_bootstrap.cc:492] T 032d30c0ca0647ab973dcdf8fd443221 P 30253ab5c6e84ceda1456c89469563b4: No bootstrap required, opened a new log
I20260925 10:44:06.956209 11293 ts_tablet_manager.cc:1397] T 032d30c0ca0647ab973dcdf8fd443221 P 30253ab5c6e84ceda1456c89469563b4: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 10:44:06.959043 11294 tablet_bootstrap.cc:492] T 032d30c0ca0647ab973dcdf8fd443221 P f321f99640ce40c69d7a5e0e48a34d8b: Bootstrap starting.
I20260925 10:44:06.959270 11295 tablet_bootstrap.cc:492] T 032d30c0ca0647ab973dcdf8fd443221 P a5eb978f48154705846247c973ebb58c: Bootstrap starting.
I20260925 10:44:06.962304 11294 tablet_bootstrap.cc:654] T 032d30c0ca0647ab973dcdf8fd443221 P f321f99640ce40c69d7a5e0e48a34d8b: Neither blocks nor log segments found. Creating new log.
I20260925 10:44:06.963001 11295 tablet_bootstrap.cc:654] T 032d30c0ca0647ab973dcdf8fd443221 P a5eb978f48154705846247c973ebb58c: Neither blocks nor log segments found. Creating new log.
I20260925 10:44:06.963441 11294 log.cc:824] T 032d30c0ca0647ab973dcdf8fd443221 P f321f99640ce40c69d7a5e0e48a34d8b: Log is configured to *not* fsync() on all Append() calls
I20260925 10:44:06.964231 11295 log.cc:824] T 032d30c0ca0647ab973dcdf8fd443221 P a5eb978f48154705846247c973ebb58c: Log is configured to *not* fsync() on all Append() calls
I20260925 10:44:06.965385 11293 raft_consensus.cc:348] T 032d30c0ca0647ab973dcdf8fd443221 P 30253ab5c6e84ceda1456c89469563b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f321f99640ce40c69d7a5e0e48a34d8b" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 45163 } } peers { permanent_uuid: "30253ab5c6e84ceda1456c89469563b4" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 42039 } } peers { permanent_uuid: "a5eb978f48154705846247c973ebb58c" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 39429 } }
I20260925 10:44:06.965919 11293 raft_consensus.cc:374] T 032d30c0ca0647ab973dcdf8fd443221 P 30253ab5c6e84ceda1456c89469563b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:44:06.966043 11293 raft_consensus.cc:729] T 032d30c0ca0647ab973dcdf8fd443221 P 30253ab5c6e84ceda1456c89469563b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30253ab5c6e84ceda1456c89469563b4, State: Initialized, Role: FOLLOWER
I20260925 10:44:06.966547 11294 tablet_bootstrap.cc:492] T 032d30c0ca0647ab973dcdf8fd443221 P f321f99640ce40c69d7a5e0e48a34d8b: No bootstrap required, opened a new log
I20260925 10:44:06.966818 11294 ts_tablet_manager.cc:1397] T 032d30c0ca0647ab973dcdf8fd443221 P f321f99640ce40c69d7a5e0e48a34d8b: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260925 10:44:06.966822 11293 consensus_queue.cc:260] T 032d30c0ca0647ab973dcdf8fd443221 P 30253ab5c6e84ceda1456c89469563b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f321f99640ce40c69d7a5e0e48a34d8b" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 45163 } } peers { permanent_uuid: "30253ab5c6e84ceda1456c89469563b4" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 42039 } } peers { permanent_uuid: "a5eb978f48154705846247c973ebb58c" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 39429 } }
I20260925 10:44:06.968081 11275 heartbeater.cc:503] Master 127.23.100.62:44381 was elected leader, sending a full tablet report...
I20260925 10:44:06.968941 11293 ts_tablet_manager.cc:1428] T 032d30c0ca0647ab973dcdf8fd443221 P 30253ab5c6e84ceda1456c89469563b4: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260925 10:44:06.971834 11295 tablet_bootstrap.cc:492] T 032d30c0ca0647ab973dcdf8fd443221 P a5eb978f48154705846247c973ebb58c: No bootstrap required, opened a new log
I20260925 10:44:06.972100 11295 ts_tablet_manager.cc:1397] T 032d30c0ca0647ab973dcdf8fd443221 P a5eb978f48154705846247c973ebb58c: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.003s
I20260925 10:44:06.975857 11294 raft_consensus.cc:348] T 032d30c0ca0647ab973dcdf8fd443221 P f321f99640ce40c69d7a5e0e48a34d8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f321f99640ce40c69d7a5e0e48a34d8b" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 45163 } } peers { permanent_uuid: "30253ab5c6e84ceda1456c89469563b4" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 42039 } } peers { permanent_uuid: "a5eb978f48154705846247c973ebb58c" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 39429 } }
I20260925 10:44:06.976212 11294 raft_consensus.cc:374] T 032d30c0ca0647ab973dcdf8fd443221 P f321f99640ce40c69d7a5e0e48a34d8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:44:06.976325 11294 raft_consensus.cc:729] T 032d30c0ca0647ab973dcdf8fd443221 P f321f99640ce40c69d7a5e0e48a34d8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f321f99640ce40c69d7a5e0e48a34d8b, State: Initialized, Role: FOLLOWER
I20260925 10:44:06.976898 11294 consensus_queue.cc:260] T 032d30c0ca0647ab973dcdf8fd443221 P f321f99640ce40c69d7a5e0e48a34d8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f321f99640ce40c69d7a5e0e48a34d8b" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 45163 } } peers { permanent_uuid: "30253ab5c6e84ceda1456c89469563b4" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 42039 } } peers { permanent_uuid: "a5eb978f48154705846247c973ebb58c" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 39429 } }
I20260925 10:44:06.978438 11294 ts_tablet_manager.cc:1428] T 032d30c0ca0647ab973dcdf8fd443221 P f321f99640ce40c69d7a5e0e48a34d8b: Time spent starting tablet: real 0.011s user 0.007s sys 0.005s
I20260925 10:44:06.980327 11295 raft_consensus.cc:348] T 032d30c0ca0647ab973dcdf8fd443221 P a5eb978f48154705846247c973ebb58c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f321f99640ce40c69d7a5e0e48a34d8b" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 45163 } } peers { permanent_uuid: "30253ab5c6e84ceda1456c89469563b4" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 42039 } } peers { permanent_uuid: "a5eb978f48154705846247c973ebb58c" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 39429 } }
I20260925 10:44:06.980746 11295 raft_consensus.cc:374] T 032d30c0ca0647ab973dcdf8fd443221 P a5eb978f48154705846247c973ebb58c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:44:06.980844 11295 raft_consensus.cc:729] T 032d30c0ca0647ab973dcdf8fd443221 P a5eb978f48154705846247c973ebb58c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5eb978f48154705846247c973ebb58c, State: Initialized, Role: FOLLOWER
I20260925 10:44:06.981428 11295 consensus_queue.cc:260] T 032d30c0ca0647ab973dcdf8fd443221 P a5eb978f48154705846247c973ebb58c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f321f99640ce40c69d7a5e0e48a34d8b" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 45163 } } peers { permanent_uuid: "30253ab5c6e84ceda1456c89469563b4" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 42039 } } peers { permanent_uuid: "a5eb978f48154705846247c973ebb58c" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 39429 } }
I20260925 10:44:06.982527 11141 heartbeater.cc:503] Master 127.23.100.62:44381 was elected leader, sending a full tablet report...
I20260925 10:44:06.983008 11295 ts_tablet_manager.cc:1428] T 032d30c0ca0647ab973dcdf8fd443221 P a5eb978f48154705846247c973ebb58c: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
W20260925 10:44:07.039705 11008 tablet.cc:2369] T 032d30c0ca0647ab973dcdf8fd443221 P f321f99640ce40c69d7a5e0e48a34d8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:44:07.069170 11142 tablet.cc:2369] T 032d30c0ca0647ab973dcdf8fd443221 P a5eb978f48154705846247c973ebb58c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:44:07.087594 11276 tablet.cc:2369] T 032d30c0ca0647ab973dcdf8fd443221 P 30253ab5c6e84ceda1456c89469563b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:44:07.172680 11300 raft_consensus.cc:482] T 032d30c0ca0647ab973dcdf8fd443221 P f321f99640ce40c69d7a5e0e48a34d8b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:44:07.173009 11300 raft_consensus.cc:504] T 032d30c0ca0647ab973dcdf8fd443221 P f321f99640ce40c69d7a5e0e48a34d8b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f321f99640ce40c69d7a5e0e48a34d8b" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 45163 } } peers { permanent_uuid: "30253ab5c6e84ceda1456c89469563b4" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 42039 } } peers { permanent_uuid: "a5eb978f48154705846247c973ebb58c" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 39429 } }
I20260925 10:44:07.174793 11300 leader_election.cc:288] T 032d30c0ca0647ab973dcdf8fd443221 P f321f99640ce40c69d7a5e0e48a34d8b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30253ab5c6e84ceda1456c89469563b4 (127.23.100.3:42039), a5eb978f48154705846247c973ebb58c (127.23.100.2:39429)
I20260925 10:44:07.181663 11292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:07.175273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:33673 (local address 127.23.100.2:39429)
0925 10:44:07.175514 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:07.175523 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:07.176394 (+ 871us) server_negotiation.cc:407] Connection header received
0925 10:44:07.176589 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:07.176592 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:07.176659 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:07.176757 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:44:07.177392 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:07.178056 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:07.179960 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:07.180724 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:07.180914 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:07.181000 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:07.181269 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 10:44:07.181395 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 10:44:07.181754 11291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:07.175326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:46637 (local address 127.23.100.3:42039)
0925 10:44:07.175544 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:07.175553 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:07.176275 (+ 722us) server_negotiation.cc:407] Connection header received
0925 10:44:07.176530 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:07.176533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:07.176593 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:07.176694 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:44:07.177324 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:07.178032 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:07.180018 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:07.180794 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:07.180991 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:07.181072 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:07.181371 (+ 299us) server_negotiation.cc:299] Negotiation successful
0925 10:44:07.181504 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 10:44:07.181821 11302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:07.175252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.2:39429 (local address 127.23.100.1:33673)
0925 10:44:07.176255 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:07.176272 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:44:07.176439 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:44:07.176775 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:44:07.176779 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:44:07.176817 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:44:07.177221 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:07.177231 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:07.178199 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:07.178205 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:07.179794 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:07.179808 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:07.180938 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:07.180944 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:07.181107 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:07.181133 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 10:44:07.181243 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 10:44:07.181366 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":893,"mutex_wait_us":23,"thread_start_us":513,"threads_started":1}
I20260925 10:44:07.182242 11303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:07.175137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.3:42039 (local address 127.23.100.1:46637)
0925 10:44:07.176147 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:07.176163 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:44:07.176345 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:44:07.176715 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:44:07.176720 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:44:07.176744 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:44:07.177177 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:07.177185 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:07.178166 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:07.178171 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:07.179855 (+ 1684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:07.179867 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:07.181057 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:07.181063 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:07.181216 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:07.181238 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 10:44:07.181343 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 10:44:07.181462 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":877,"mutex_wait_us":71,"thread_start_us":757,"threads_started":1}
I20260925 10:44:07.182849 11096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "032d30c0ca0647ab973dcdf8fd443221" candidate_uuid: "f321f99640ce40c69d7a5e0e48a34d8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5eb978f48154705846247c973ebb58c" is_pre_election: true
I20260925 10:44:07.183182 11230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "032d30c0ca0647ab973dcdf8fd443221" candidate_uuid: "f321f99640ce40c69d7a5e0e48a34d8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30253ab5c6e84ceda1456c89469563b4" is_pre_election: true
I20260925 10:44:07.183364 11096 raft_consensus.cc:2454] T 032d30c0ca0647ab973dcdf8fd443221 P a5eb978f48154705846247c973ebb58c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f321f99640ce40c69d7a5e0e48a34d8b in term 0.
I20260925 10:44:07.183655 11230 raft_consensus.cc:2454] T 032d30c0ca0647ab973dcdf8fd443221 P 30253ab5c6e84ceda1456c89469563b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f321f99640ce40c69d7a5e0e48a34d8b in term 0.
I20260925 10:44:07.184139 10897 leader_election.cc:302] T 032d30c0ca0647ab973dcdf8fd443221 P f321f99640ce40c69d7a5e0e48a34d8b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5eb978f48154705846247c973ebb58c, f321f99640ce40c69d7a5e0e48a34d8b; no voters:
I20260925 10:44:07.184543 11300 raft_consensus.cc:2781] T 032d30c0ca0647ab973dcdf8fd443221 P f321f99640ce40c69d7a5e0e48a34d8b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:44:07.184696 11300 raft_consensus.cc:482] T 032d30c0ca0647ab973dcdf8fd443221 P f321f99640ce40c69d7a5e0e48a34d8b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:44:07.184806 11300 raft_consensus.cc:3037] T 032d30c0ca0647ab973dcdf8fd443221 P f321f99640ce40c69d7a5e0e48a34d8b [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:44:07.186992 11300 raft_consensus.cc:504] T 032d30c0ca0647ab973dcdf8fd443221 P f321f99640ce40c69d7a5e0e48a34d8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f321f99640ce40c69d7a5e0e48a34d8b" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 45163 } } peers { permanent_uuid: "30253ab5c6e84ceda1456c89469563b4" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 42039 } } peers { permanent_uuid: "a5eb978f48154705846247c973ebb58c" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 39429 } }
I20260925 10:44:07.187566 11300 leader_election.cc:288] T 032d30c0ca0647ab973dcdf8fd443221 P f321f99640ce40c69d7a5e0e48a34d8b [CANDIDATE]: Term 1 election: Requested vote from peers 30253ab5c6e84ceda1456c89469563b4 (127.23.100.3:42039), a5eb978f48154705846247c973ebb58c (127.23.100.2:39429)
I20260925 10:44:07.187939 11230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "032d30c0ca0647ab973dcdf8fd443221" candidate_uuid: "f321f99640ce40c69d7a5e0e48a34d8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30253ab5c6e84ceda1456c89469563b4"
I20260925 10:44:07.187981 11096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "032d30c0ca0647ab973dcdf8fd443221" candidate_uuid: "f321f99640ce40c69d7a5e0e48a34d8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5eb978f48154705846247c973ebb58c"
I20260925 10:44:07.188141 11230 raft_consensus.cc:3037] T 032d30c0ca0647ab973dcdf8fd443221 P 30253ab5c6e84ceda1456c89469563b4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:44:07.188180 11096 raft_consensus.cc:3037] T 032d30c0ca0647ab973dcdf8fd443221 P a5eb978f48154705846247c973ebb58c [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:44:07.190274 11230 raft_consensus.cc:2454] T 032d30c0ca0647ab973dcdf8fd443221 P 30253ab5c6e84ceda1456c89469563b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f321f99640ce40c69d7a5e0e48a34d8b in term 1.
I20260925 10:44:07.190587 11096 raft_consensus.cc:2454] T 032d30c0ca0647ab973dcdf8fd443221 P a5eb978f48154705846247c973ebb58c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f321f99640ce40c69d7a5e0e48a34d8b in term 1.
I20260925 10:44:07.190790 10898 leader_election.cc:302] T 032d30c0ca0647ab973dcdf8fd443221 P f321f99640ce40c69d7a5e0e48a34d8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30253ab5c6e84ceda1456c89469563b4, f321f99640ce40c69d7a5e0e48a34d8b; no voters:
I20260925 10:44:07.191102 11300 raft_consensus.cc:2781] T 032d30c0ca0647ab973dcdf8fd443221 P f321f99640ce40c69d7a5e0e48a34d8b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:44:07.191521 11300 raft_consensus.cc:686] T 032d30c0ca0647ab973dcdf8fd443221 P f321f99640ce40c69d7a5e0e48a34d8b [term 1 LEADER]: Becoming Leader. State: Replica: f321f99640ce40c69d7a5e0e48a34d8b, State: Running, Role: LEADER
I20260925 10:44:07.192090 11300 consensus_queue.cc:237] T 032d30c0ca0647ab973dcdf8fd443221 P f321f99640ce40c69d7a5e0e48a34d8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f321f99640ce40c69d7a5e0e48a34d8b" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 45163 } } peers { permanent_uuid: "30253ab5c6e84ceda1456c89469563b4" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 42039 } } peers { permanent_uuid: "a5eb978f48154705846247c973ebb58c" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 39429 } }
I20260925 10:44:07.196396 10817 catalog_manager.cc:5495] T 032d30c0ca0647ab973dcdf8fd443221 P f321f99640ce40c69d7a5e0e48a34d8b reported cstate change: term changed from 0 to 1, leader changed from <none> to f321f99640ce40c69d7a5e0e48a34d8b (127.23.100.1). New cstate: current_term: 1 leader_uuid: "f321f99640ce40c69d7a5e0e48a34d8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f321f99640ce40c69d7a5e0e48a34d8b" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 45163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30253ab5c6e84ceda1456c89469563b4" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 42039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5eb978f48154705846247c973ebb58c" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 39429 } health_report { overall_health: UNKNOWN } } }
I20260925 10:44:07.225064 11288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:07.220478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50796 (local address 127.23.100.1:45163)
0925 10:44:07.220674 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:07.220679 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:07.220694 (+ 15us) server_negotiation.cc:407] Connection header received
0925 10:44:07.220776 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:07.220779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:07.220866 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:07.220985 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:44:07.221300 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:07.221995 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:07.223285 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:07.223662 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:07.223751 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:07.224138 (+ 387us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:44:07.224629 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:44:07.224731 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:07.224818 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 10:44:07.224880 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 10:44:07.233006 11096 raft_consensus.cc:1261] T 032d30c0ca0647ab973dcdf8fd443221 P a5eb978f48154705846247c973ebb58c [term 1 FOLLOWER]: Refusing update from remote peer f321f99640ce40c69d7a5e0e48a34d8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:44:07.232992 11230 raft_consensus.cc:1261] T 032d30c0ca0647ab973dcdf8fd443221 P 30253ab5c6e84ceda1456c89469563b4 [term 1 FOLLOWER]: Refusing update from remote peer f321f99640ce40c69d7a5e0e48a34d8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:44:07.233953 11304 consensus_queue.cc:1035] T 032d30c0ca0647ab973dcdf8fd443221 P f321f99640ce40c69d7a5e0e48a34d8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5eb978f48154705846247c973ebb58c" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 39429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:44:07.234365 11300 consensus_queue.cc:1035] T 032d30c0ca0647ab973dcdf8fd443221 P f321f99640ce40c69d7a5e0e48a34d8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "30253ab5c6e84ceda1456c89469563b4" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 42039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:44:07.252380 11309 mvcc.cc:205] Tried to move back new op lower bound from 7333204161456357376 to 7333204161302335488. Current Snapshot: MvccSnapshot[applied={T|T < 7333204161456357376}]
I20260925 10:44:07.262360 10817 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:44:07.262677 10817 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:44:07.266412 10817 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7b5fda6815ff4a139d4a1ef4e5be537b: Sending DeleteTablet for 3 replicas of tablet 032d30c0ca0647ab973dcdf8fd443221
I20260925 10:44:07.267498 10942 tablet_service.cc:1514] Processing DeleteTablet for tablet 032d30c0ca0647ab973dcdf8fd443221 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:44:07 UTC) from {username='kudu', principal='kudu/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:50790
I20260925 10:44:07.267663 11076 tablet_service.cc:1514] Processing DeleteTablet for tablet 032d30c0ca0647ab973dcdf8fd443221 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:44:07 UTC) from {username='kudu', principal='kudu/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:54908
I20260925 10:44:07.268226 11210 tablet_service.cc:1514] Processing DeleteTablet for tablet 032d30c0ca0647ab973dcdf8fd443221 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:44:07 UTC) from {username='kudu', principal='kudu/127.23.100.62@KRBTEST.COM'} at 127.0.0.1:54440
I20260925 10:44:07.269325 11318 tablet_replica.cc:330] T 032d30c0ca0647ab973dcdf8fd443221 P 30253ab5c6e84ceda1456c89469563b4: stopping tablet replica
I20260925 10:44:07.269740 11318 raft_consensus.cc:2229] T 032d30c0ca0647ab973dcdf8fd443221 P 30253ab5c6e84ceda1456c89469563b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:44:07.270238 11318 raft_consensus.cc:2258] T 032d30c0ca0647ab973dcdf8fd443221 P 30253ab5c6e84ceda1456c89469563b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:44:07.271035 11317 tablet_replica.cc:330] T 032d30c0ca0647ab973dcdf8fd443221 P a5eb978f48154705846247c973ebb58c: stopping tablet replica
I20260925 10:44:07.271450 11317 raft_consensus.cc:2229] T 032d30c0ca0647ab973dcdf8fd443221 P a5eb978f48154705846247c973ebb58c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:44:07.271950 11317 raft_consensus.cc:2258] T 032d30c0ca0647ab973dcdf8fd443221 P a5eb978f48154705846247c973ebb58c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:44:07.272744 11318 ts_tablet_manager.cc:1905] T 032d30c0ca0647ab973dcdf8fd443221 P 30253ab5c6e84ceda1456c89469563b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:44:07.273996 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 10879
I20260925 10:44:07.274021 11317 ts_tablet_manager.cc:1905] T 032d30c0ca0647ab973dcdf8fd443221 P a5eb978f48154705846247c973ebb58c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:44:07.280503 11316 tablet_replica.cc:330] T 032d30c0ca0647ab973dcdf8fd443221 P f321f99640ce40c69d7a5e0e48a34d8b: stopping tablet replica
I20260925 10:44:07.280923 11316 raft_consensus.cc:2229] T 032d30c0ca0647ab973dcdf8fd443221 P f321f99640ce40c69d7a5e0e48a34d8b [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:44:07.281602 11316 raft_consensus.cc:2258] T 032d30c0ca0647ab973dcdf8fd443221 P f321f99640ce40c69d7a5e0e48a34d8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:44:07.281985 11317 ts_tablet_manager.cc:1918] T 032d30c0ca0647ab973dcdf8fd443221 P a5eb978f48154705846247c973ebb58c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:44:07.282198 11317 log.cc:1196] T 032d30c0ca0647ab973dcdf8fd443221 P a5eb978f48154705846247c973ebb58c: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/ts-1/wal/wals/032d30c0ca0647ab973dcdf8fd443221
I20260925 10:44:07.282938 11317 ts_tablet_manager.cc:1939] T 032d30c0ca0647ab973dcdf8fd443221 P a5eb978f48154705846247c973ebb58c: Deleting consensus metadata
I20260925 10:44:07.283794 11316 ts_tablet_manager.cc:1905] T 032d30c0ca0647ab973dcdf8fd443221 P f321f99640ce40c69d7a5e0e48a34d8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:44:07.285080 10804 catalog_manager.cc:4841] TS a5eb978f48154705846247c973ebb58c (127.23.100.2:39429): tablet 032d30c0ca0647ab973dcdf8fd443221 (table test-table [id=fd0117ff8e98470192a0b3994aef480a]) successfully deleted
I20260925 10:44:07.288362 11318 ts_tablet_manager.cc:1918] T 032d30c0ca0647ab973dcdf8fd443221 P 30253ab5c6e84ceda1456c89469563b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:44:07.288528 11318 log.cc:1196] T 032d30c0ca0647ab973dcdf8fd443221 P 30253ab5c6e84ceda1456c89469563b4: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/ts-2/wal/wals/032d30c0ca0647ab973dcdf8fd443221
I20260925 10:44:07.289127 11318 ts_tablet_manager.cc:1939] T 032d30c0ca0647ab973dcdf8fd443221 P 30253ab5c6e84ceda1456c89469563b4: Deleting consensus metadata
I20260925 10:44:07.291119 11316 ts_tablet_manager.cc:1918] T 032d30c0ca0647ab973dcdf8fd443221 P f321f99640ce40c69d7a5e0e48a34d8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:44:07.291273 11316 log.cc:1196] T 032d30c0ca0647ab973dcdf8fd443221 P f321f99640ce40c69d7a5e0e48a34d8b: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790332740062128-23952-0/minicluster-data/ts-0/wal/wals/032d30c0ca0647ab973dcdf8fd443221
I20260925 10:44:07.291252 10805 catalog_manager.cc:4841] TS 30253ab5c6e84ceda1456c89469563b4 (127.23.100.3:42039): tablet 032d30c0ca0647ab973dcdf8fd443221 (table test-table [id=fd0117ff8e98470192a0b3994aef480a]) successfully deleted
I20260925 10:44:07.291854 11316 ts_tablet_manager.cc:1939] T 032d30c0ca0647ab973dcdf8fd443221 P f321f99640ce40c69d7a5e0e48a34d8b: Deleting consensus metadata
I20260925 10:44:07.293524 10803 catalog_manager.cc:4841] TS f321f99640ce40c69d7a5e0e48a34d8b (127.23.100.1:45163): tablet 032d30c0ca0647ab973dcdf8fd443221 (table test-table [id=fd0117ff8e98470192a0b3994aef480a]) successfully deleted
Sep 25 10:44:07 dist-test-slave-dxdd krb5kdc[10773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.1@KRBTEST.COM
I20260925 10:44:07.307832 11288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:07.276816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50810 (local address 127.23.100.1:45163)
0925 10:44:07.276985 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:07.276991 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:07.292743 (+ 15752us) server_negotiation.cc:407] Connection header received
0925 10:44:07.292957 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:07.292963 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:07.293046 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:07.293147 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:07.294500 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:07.295304 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:07.296515 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:07.296964 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:07.297112 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:07.301782 (+ 4670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:07.301795 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:07.301808 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:44:07.301848 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:07.304872 (+ 3024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:07.305726 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:07.305733 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:07.305735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:07.305867 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:07.306297 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:07.306300 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:07.306378 (+ 78us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:07.306731 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:07.306953 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:07.307337 (+ 384us) server_negotiation.cc:299] Negotiation successful
0925 10:44:07.307509 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 10:44:07.308931 11002 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:44:07.407898 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 10879
I20260925 10:44:07.426397 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 11013
Sep 25 10:44:07 dist-test-slave-dxdd krb5kdc[10773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.2@KRBTEST.COM
I20260925 10:44:07.441109 11292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:07.427187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54922 (local address 127.23.100.2:39429)
0925 10:44:07.427363 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:07.427367 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:07.427385 (+ 18us) server_negotiation.cc:407] Connection header received
0925 10:44:07.427569 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:07.427573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:07.427635 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:07.427722 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:07.429009 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:07.429736 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:07.431089 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:07.431469 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:07.431561 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:07.435975 (+ 4414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:07.435987 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:07.436003 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:44:07.436038 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:07.438750 (+ 2712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:07.439373 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:07.439379 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:07.439382 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:07.439505 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:07.439813 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:07.439816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:07.439818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:07.440239 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:07.440541 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:07.440705 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 10:44:07.440858 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 10:44:07.441744 11136 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:44:07.530519 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 11013
I20260925 10:44:07.547307 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 11147
Sep 25 10:44:07 dist-test-slave-dxdd krb5kdc[10773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.3@KRBTEST.COM
I20260925 10:44:07.562229 11291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:07.548056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54442 (local address 127.23.100.3:42039)
0925 10:44:07.548242 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:07.548247 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:07.548304 (+ 57us) server_negotiation.cc:407] Connection header received
0925 10:44:07.548446 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:07.548450 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:07.548514 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:07.548598 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:07.549915 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:07.550717 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:07.552082 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:07.552502 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:07.552593 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:07.556924 (+ 4331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:07.556938 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:07.556954 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:44:07.556988 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:07.559548 (+ 2560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:07.560237 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:07.560245 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:07.560247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:07.560375 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:07.560768 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:07.560773 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:07.560776 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:07.561099 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:07.561390 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:07.561689 (+ 299us) server_negotiation.cc:299] Negotiation successful
0925 10:44:07.561896 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 10:44:07.562870 11270 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:44:07.649125 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 11147
I20260925 10:44:07.665400 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 10786
I20260925 10:44:07.666510 10848 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:44:07.776250 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 10786
2026-09-25T10:44:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4323 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T10:44:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:44:07Z Disabled control of system clock
I20260925 10:44:07.851413 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:44957
--webserver_interface=127.23.100.62
--webserver_port=0
--builtin_ntp_servers=127.23.100.20:41465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.62:44957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 10:44:08.172163 11327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:44:08.172454 11327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:44:08.172533 11327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:44:08.179263 11327 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:44:08.179356 11327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:44:08.179394 11327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:44:08.179452 11327 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:44:08.179495 11327 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:44:08.188084 11327 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:41465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.62:44957
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:44957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:44:08.189267 11327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:44:08.190923 11327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:44:08.202898 11332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:08.202859 11335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:08.202936 11333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:44:08.203958 11327 server_base.cc:1034] running on GCE node
I20260925 10:44:08.204591 11327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:44:08.205834 11327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:44:08.207023 11327 hybrid_clock.cc:648] HybridClock initialized: now 1790333048206984 us; error 34 us; skew 500 ppm
I20260925 10:44:08.207360 11327 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:44:08.209503 11327 webserver.cc:460] Webserver started at http://127.23.100.62:36713/ using document root <none> and password file <none>
I20260925 10:44:08.210096 11327 fs_manager.cc:359] Metadata directory not provided
I20260925 10:44:08.210180 11327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:44:08.210520 11327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:44:08.213294 11327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/master-0/data/instance:
uuid: "5a557e9b349c4a53a4a656488694f36e"
format_stamp: "Formatted at 2026-09-25 10:44:08 on dist-test-slave-dxdd"
server_key: "a6ac156e9331c2300c680dba8de5a441"
server_key_iv: "f1784ce87eeb7a893d98eb55016120f3"
server_key_version: "encryptionkey@0"
I20260925 10:44:08.213996 11327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/master-0/wal/instance:
uuid: "5a557e9b349c4a53a4a656488694f36e"
format_stamp: "Formatted at 2026-09-25 10:44:08 on dist-test-slave-dxdd"
server_key: "a6ac156e9331c2300c680dba8de5a441"
server_key_iv: "f1784ce87eeb7a893d98eb55016120f3"
server_key_version: "encryptionkey@0"
I20260925 10:44:08.218705 11327 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260925 10:44:08.221755 11341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:44:08.222996 11327 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 10:44:08.223136 11327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/master-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/master-0/wal
uuid: "5a557e9b349c4a53a4a656488694f36e"
format_stamp: "Formatted at 2026-09-25 10:44:08 on dist-test-slave-dxdd"
server_key: "a6ac156e9331c2300c680dba8de5a441"
server_key_iv: "f1784ce87eeb7a893d98eb55016120f3"
server_key_version: "encryptionkey@0"
I20260925 10:44:08.223299 11327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:44:08.254479 11327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:44:08.255254 11327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:44:08.255482 11327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:44:08.273387 11327 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.62:44957
I20260925 10:44:08.273406 11392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.62:44957 every 8 connection(s)
I20260925 10:44:08.274746 11327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
I20260925 10:44:08.278106 11393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:44:08.282729 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 11327
I20260925 10:44:08.282891 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/master-0/wal/instance
I20260925 10:44:08.283356 23952 external_mini_cluster.cc:1428] Setting key 8c863f44b91be81a26422790a7cf8e6b
I20260925 10:44:08.288635 11393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a557e9b349c4a53a4a656488694f36e: Bootstrap starting.
I20260925 10:44:08.291818 11393 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a557e9b349c4a53a4a656488694f36e: Neither blocks nor log segments found. Creating new log.
I20260925 10:44:08.292762 11393 log.cc:824] T 00000000000000000000000000000000 P 5a557e9b349c4a53a4a656488694f36e: Log is configured to *not* fsync() on all Append() calls
I20260925 10:44:08.295526 11393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a557e9b349c4a53a4a656488694f36e: No bootstrap required, opened a new log
I20260925 10:44:08.297392 11396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:08.285115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38536 (local address 127.23.100.62:44957)
0925 10:44:08.286175 (+ 1060us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:08.286188 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:08.286232 (+ 44us) server_negotiation.cc:407] Connection header received
0925 10:44:08.288721 (+ 2489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:08.288747 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:08.289410 (+ 663us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:08.289864 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:08.290572 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:08.291831 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:08.293229 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:08.293908 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:08.294133 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:08.295253 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:08.295260 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:08.295273 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:44:08.295814 (+ 541us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:08.295892 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:44:08.295912 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:08.296047 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:08.296268 (+ 221us) server_negotiation.cc:299] Negotiation successful
0925 10:44:08.296543 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":413,"threads_started":1}
I20260925 10:44:08.303552 11393 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a557e9b349c4a53a4a656488694f36e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a557e9b349c4a53a4a656488694f36e" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 44957 } }
I20260925 10:44:08.303896 11393 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a557e9b349c4a53a4a656488694f36e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:44:08.304000 11393 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a557e9b349c4a53a4a656488694f36e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a557e9b349c4a53a4a656488694f36e, State: Initialized, Role: FOLLOWER
I20260925 10:44:08.304510 11393 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a557e9b349c4a53a4a656488694f36e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a557e9b349c4a53a4a656488694f36e" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 44957 } }
I20260925 10:44:08.304781 11393 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a557e9b349c4a53a4a656488694f36e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:44:08.304921 11393 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a557e9b349c4a53a4a656488694f36e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:44:08.305135 11393 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a557e9b349c4a53a4a656488694f36e [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:44:08.307482 11393 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a557e9b349c4a53a4a656488694f36e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a557e9b349c4a53a4a656488694f36e" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 44957 } }
I20260925 10:44:08.307929 11393 leader_election.cc:302] T 00000000000000000000000000000000 P 5a557e9b349c4a53a4a656488694f36e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a557e9b349c4a53a4a656488694f36e; no voters:
I20260925 10:44:08.308552 11393 leader_election.cc:288] T 00000000000000000000000000000000 P 5a557e9b349c4a53a4a656488694f36e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:44:08.308730 11398 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a557e9b349c4a53a4a656488694f36e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:44:08.309464 11398 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a557e9b349c4a53a4a656488694f36e [term 1 LEADER]: Becoming Leader. State: Replica: 5a557e9b349c4a53a4a656488694f36e, State: Running, Role: LEADER
I20260925 10:44:08.309948 11398 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a557e9b349c4a53a4a656488694f36e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a557e9b349c4a53a4a656488694f36e" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 44957 } }
I20260925 10:44:08.310592 11393 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a557e9b349c4a53a4a656488694f36e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:44:08.312889 11399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a557e9b349c4a53a4a656488694f36e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a557e9b349c4a53a4a656488694f36e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a557e9b349c4a53a4a656488694f36e" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 44957 } } }
I20260925 10:44:08.314903 11399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a557e9b349c4a53a4a656488694f36e [sys.catalog]: This master's current role is: LEADER
I20260925 10:44:08.312844 11400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a557e9b349c4a53a4a656488694f36e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a557e9b349c4a53a4a656488694f36e. Latest consensus state: current_term: 1 leader_uuid: "5a557e9b349c4a53a4a656488694f36e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a557e9b349c4a53a4a656488694f36e" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 44957 } } }
I20260925 10:44:08.315423 11400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a557e9b349c4a53a4a656488694f36e [sys.catalog]: This master's current role is: LEADER
I20260925 10:44:08.319482 11407 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:44:08.324110 11407 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:44:08.332180 11407 catalog_manager.cc:1347] Generated new cluster ID: fae9c544c4d748e5b19ae5a92dfeb312
I20260925 10:44:08.332296 11407 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:44:08.350643 11407 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:44:08.352283 11407 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:44:08.369515 11407 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a557e9b349c4a53a4a656488694f36e: Generated new TSK 0
I20260925 10:44:08.370538 11407 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:44:08.378716 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.1:0
--local_ip_for_outbound_sockets=127.23.100.1
--webserver_interface=127.23.100.1
--webserver_port=0
--tserver_master_addrs=127.23.100.62:44957
--builtin_ntp_servers=127.23.100.20:41465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 10:44:08.704656 11417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:44:08.704955 11417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:44:08.705123 11417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:44:08.712303 11417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:44:08.712420 11417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:44:08.712586 11417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.1
I20260925 10:44:08.721774 11417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:41465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.1
--webserver_port=0
--tserver_master_addrs=127.23.100.62:44957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.1
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:44:08.723171 11417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:44:08.724969 11417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:44:08.738387 11422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:08.738447 11423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:08.738639 11425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:44:08.739481 11417 server_base.cc:1034] running on GCE node
I20260925 10:44:08.740044 11417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:44:08.740914 11417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:44:08.742153 11417 hybrid_clock.cc:648] HybridClock initialized: now 1790333048742107 us; error 54 us; skew 500 ppm
I20260925 10:44:08.742518 11417 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:44:08.744628 11417 webserver.cc:460] Webserver started at http://127.23.100.1:44483/ using document root <none> and password file <none>
I20260925 10:44:08.745288 11417 fs_manager.cc:359] Metadata directory not provided
I20260925 10:44:08.745419 11417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:44:08.745736 11417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:44:08.748505 11417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/ts-0/data/instance:
uuid: "c124058dd1474b8c94f4c791ad5bf131"
format_stamp: "Formatted at 2026-09-25 10:44:08 on dist-test-slave-dxdd"
server_key: "c2e4dabc6ea5623beef937b86f516648"
server_key_iv: "18278a5908ec1719877704814dcaa8ac"
server_key_version: "encryptionkey@0"
I20260925 10:44:08.749265 11417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/ts-0/wal/instance:
uuid: "c124058dd1474b8c94f4c791ad5bf131"
format_stamp: "Formatted at 2026-09-25 10:44:08 on dist-test-slave-dxdd"
server_key: "c2e4dabc6ea5623beef937b86f516648"
server_key_iv: "18278a5908ec1719877704814dcaa8ac"
server_key_version: "encryptionkey@0"
I20260925 10:44:08.754096 11417 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260925 10:44:08.757262 11431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:44:08.758556 11417 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 10:44:08.758723 11417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/ts-0/wal
uuid: "c124058dd1474b8c94f4c791ad5bf131"
format_stamp: "Formatted at 2026-09-25 10:44:08 on dist-test-slave-dxdd"
server_key: "c2e4dabc6ea5623beef937b86f516648"
server_key_iv: "18278a5908ec1719877704814dcaa8ac"
server_key_version: "encryptionkey@0"
I20260925 10:44:08.758942 11417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:44:08.770534 11417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:44:08.771299 11417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:44:08.771600 11417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:44:08.772470 11417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:44:08.773850 11417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:44:08.773968 11417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:44:08.774098 11417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:44:08.774185 11417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:44:08.806777 11417 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.1:46759
I20260925 10:44:08.806828 11543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.1:46759 every 8 connection(s)
I20260925 10:44:08.808059 11417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/ts-0/data/info.pb
I20260925 10:44:08.815951 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 11417
I20260925 10:44:08.816145 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/ts-0/wal/instance
I20260925 10:44:08.816574 23952 external_mini_cluster.cc:1428] Setting key e8cef096448f4811c4d31d92457b4c62
I20260925 10:44:08.819304 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.2:0
--local_ip_for_outbound_sockets=127.23.100.2
--webserver_interface=127.23.100.2
--webserver_port=0
--tserver_master_addrs=127.23.100.62:44957
--builtin_ntp_servers=127.23.100.20:41465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 10:44:08.821012 11546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:08.810563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.1:47929 (local address 127.23.100.62:44957)
0925 10:44:08.811029 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:08.811033 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:08.812294 (+ 1261us) server_negotiation.cc:407] Connection header received
0925 10:44:08.813994 (+ 1700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:08.813999 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:08.814094 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:08.814215 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:08.815204 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:08.816299 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:08.818141 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:08.818649 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:08.818782 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:08.820161 (+ 1379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:08.820167 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:08.820171 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:44:08.820196 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:08.820237 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:44:08.820249 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:08.820343 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:08.820576 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 10:44:08.820672 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":259,"threads_started":1}
I20260925 10:44:08.821933 11547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:08.811203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44957 (local address 127.23.100.1:47929)
0925 10:44:08.812146 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:08.812179 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 10:44:08.813739 (+ 1560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:44:08.814454 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:44:08.814461 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:44:08.814545 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:44:08.815037 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:08.815061 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:08.816492 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:08.816500 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:08.817988 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:08.818000 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:08.818882 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:08.818890 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:08.819165 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:08.819828 (+ 663us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:44:08.819830 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:44:08.819873 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:44:08.819945 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:44:08.820356 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:44:08.820369 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:44:08.820377 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 10:44:08.820564 (+ 187us) client_negotiation.cc:240] Negotiation successful
0925 10:44:08.820937 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":296,"threads_started":1}
I20260925 10:44:08.823462 11544 heartbeater.cc:347] Connected to a master server at 127.23.100.62:44957
I20260925 10:44:08.823755 11544 heartbeater.cc:464] Registering TS with master...
I20260925 10:44:08.824645 11544 heartbeater.cc:511] Master 127.23.100.62:44957 requested a full tablet report, sending...
I20260925 10:44:08.826828 11358 ts_manager.cc:194] Registered new tserver with Master: c124058dd1474b8c94f4c791ad5bf131 (127.23.100.1:46759)
I20260925 10:44:08.828341 11358 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.100.1:47929
W20260925 10:44:09.143239 11548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:44:09.143487 11548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:44:09.143646 11548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:44:09.150717 11548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:44:09.150830 11548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:44:09.151000 11548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.2
I20260925 10:44:09.159657 11548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:41465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.2
--webserver_port=0
--tserver_master_addrs=127.23.100.62:44957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.2
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:44:09.160880 11548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:44:09.162700 11548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:44:09.175549 11553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:09.175583 11556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:09.175633 11554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:44:09.176409 11548 server_base.cc:1034] running on GCE node
I20260925 10:44:09.176911 11548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:44:09.177722 11548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:44:09.178910 11548 hybrid_clock.cc:648] HybridClock initialized: now 1790333049178856 us; error 48 us; skew 500 ppm
I20260925 10:44:09.179229 11548 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:44:09.181416 11548 webserver.cc:460] Webserver started at http://127.23.100.2:40813/ using document root <none> and password file <none>
I20260925 10:44:09.182096 11548 fs_manager.cc:359] Metadata directory not provided
I20260925 10:44:09.182217 11548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:44:09.182549 11548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:44:09.185235 11548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/ts-1/data/instance:
uuid: "e43df83593b044a482eb86538ee7a758"
format_stamp: "Formatted at 2026-09-25 10:44:09 on dist-test-slave-dxdd"
server_key: "705e684f8f863778eb55e9170acc9d7f"
server_key_iv: "0a2d3bfdd4fb35482fa2e2184c92d78a"
server_key_version: "encryptionkey@0"
I20260925 10:44:09.185987 11548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/ts-1/wal/instance:
uuid: "e43df83593b044a482eb86538ee7a758"
format_stamp: "Formatted at 2026-09-25 10:44:09 on dist-test-slave-dxdd"
server_key: "705e684f8f863778eb55e9170acc9d7f"
server_key_iv: "0a2d3bfdd4fb35482fa2e2184c92d78a"
server_key_version: "encryptionkey@0"
I20260925 10:44:09.190860 11548 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 10:44:09.194069 11562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:44:09.195538 11548 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 10:44:09.195729 11548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/ts-1/wal
uuid: "e43df83593b044a482eb86538ee7a758"
format_stamp: "Formatted at 2026-09-25 10:44:09 on dist-test-slave-dxdd"
server_key: "705e684f8f863778eb55e9170acc9d7f"
server_key_iv: "0a2d3bfdd4fb35482fa2e2184c92d78a"
server_key_version: "encryptionkey@0"
I20260925 10:44:09.195938 11548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:44:09.219601 11548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:44:09.220369 11548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:44:09.220671 11548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:44:09.221585 11548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:44:09.223031 11548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:44:09.223165 11548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:44:09.223284 11548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:44:09.223376 11548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:44:09.257121 11548 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.2:46479
I20260925 10:44:09.257241 11674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.2:46479 every 8 connection(s)
I20260925 10:44:09.258467 11548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/ts-1/data/info.pb
I20260925 10:44:09.264124 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 11548
I20260925 10:44:09.264299 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/ts-1/wal/instance
I20260925 10:44:09.264928 23952 external_mini_cluster.cc:1428] Setting key 5a744265a5ac1d52c17fc33d20e6b755
I20260925 10:44:09.267978 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.3:0
--local_ip_for_outbound_sockets=127.23.100.3
--webserver_interface=127.23.100.3
--webserver_port=0
--tserver_master_addrs=127.23.100.62:44957
--builtin_ntp_servers=127.23.100.20:41465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 10:44:09.272354 11546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:09.260898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:50537 (local address 127.23.100.62:44957)
0925 10:44:09.261122 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:09.261128 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:09.262578 (+ 1450us) server_negotiation.cc:407] Connection header received
0925 10:44:09.264808 (+ 2230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:09.264812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:09.264878 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:09.264965 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:09.266197 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:09.267213 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:09.269700 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:09.270072 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:09.270173 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:09.271681 (+ 1508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:09.271685 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:09.271688 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:44:09.271704 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:09.271735 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:44:09.271746 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:09.271815 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:09.272054 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 10:44:09.272123 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260925 10:44:09.273731 11677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:09.261434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44957 (local address 127.23.100.2:50537)
0925 10:44:09.262398 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:09.262468 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0925 10:44:09.264515 (+ 2047us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:44:09.265237 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:44:09.265248 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:44:09.265365 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:44:09.266023 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:09.266057 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:09.267375 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:09.267380 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:09.269561 (+ 2181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:09.269579 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:09.270286 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:09.270295 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:09.270655 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:09.271345 (+ 690us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:44:09.271351 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:44:09.271403 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:44:09.271500 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:44:09.271842 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:44:09.271854 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:44:09.271865 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 10:44:09.272036 (+ 171us) client_negotiation.cc:240] Negotiation successful
0925 10:44:09.272438 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":311,"threads_started":1}
I20260925 10:44:09.275450 11675 heartbeater.cc:347] Connected to a master server at 127.23.100.62:44957
I20260925 10:44:09.275758 11675 heartbeater.cc:464] Registering TS with master...
I20260925 10:44:09.276564 11675 heartbeater.cc:511] Master 127.23.100.62:44957 requested a full tablet report, sending...
I20260925 10:44:09.278060 11358 ts_manager.cc:194] Registered new tserver with Master: e43df83593b044a482eb86538ee7a758 (127.23.100.2:46479)
I20260925 10:44:09.278955 11358 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.100.2:50537
W20260925 10:44:09.590039 11678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:44:09.590348 11678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:44:09.590456 11678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:44:09.597438 11678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:44:09.597522 11678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:44:09.597620 11678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.3
I20260925 10:44:09.607107 11678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:41465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.3
--webserver_port=0
--tserver_master_addrs=127.23.100.62:44957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.3
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:44:09.608263 11678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:44:09.609882 11678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:44:09.623008 11683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:09.622939 11684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:09.622982 11686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:44:09.623737 11678 server_base.cc:1034] running on GCE node
I20260925 10:44:09.624286 11678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:44:09.625226 11678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:44:09.626396 11678 hybrid_clock.cc:648] HybridClock initialized: now 1790333049626356 us; error 32 us; skew 500 ppm
I20260925 10:44:09.626753 11678 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:44:09.629004 11678 webserver.cc:460] Webserver started at http://127.23.100.3:40627/ using document root <none> and password file <none>
I20260925 10:44:09.629668 11678 fs_manager.cc:359] Metadata directory not provided
I20260925 10:44:09.629797 11678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:44:09.630089 11678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:44:09.632781 11678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/ts-2/data/instance:
uuid: "7995998859f44db9a61f75c7478f0f51"
format_stamp: "Formatted at 2026-09-25 10:44:09 on dist-test-slave-dxdd"
server_key: "83378f7ea0ea669d5c080b60b4a66cd5"
server_key_iv: "1f0cc9d69b3e0d077f56dda36953040c"
server_key_version: "encryptionkey@0"
I20260925 10:44:09.633523 11678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/ts-2/wal/instance:
uuid: "7995998859f44db9a61f75c7478f0f51"
format_stamp: "Formatted at 2026-09-25 10:44:09 on dist-test-slave-dxdd"
server_key: "83378f7ea0ea669d5c080b60b4a66cd5"
server_key_iv: "1f0cc9d69b3e0d077f56dda36953040c"
server_key_version: "encryptionkey@0"
I20260925 10:44:09.638180 11678 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260925 10:44:09.641304 11692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:44:09.642534 11678 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 10:44:09.642699 11678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/ts-2/wal
uuid: "7995998859f44db9a61f75c7478f0f51"
format_stamp: "Formatted at 2026-09-25 10:44:09 on dist-test-slave-dxdd"
server_key: "83378f7ea0ea669d5c080b60b4a66cd5"
server_key_iv: "1f0cc9d69b3e0d077f56dda36953040c"
server_key_version: "encryptionkey@0"
I20260925 10:44:09.642901 11678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:44:09.658211 11678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:44:09.658933 11678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:44:09.659245 11678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:44:09.660120 11678 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:44:09.661398 11678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:44:09.661520 11678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:44:09.661641 11678 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:44:09.661726 11678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:44:09.693650 11678 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.3:34873
I20260925 10:44:09.693737 11805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.3:34873 every 8 connection(s)
I20260925 10:44:09.694898 11678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/ts-2/data/info.pb
I20260925 10:44:09.699030 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 11678
I20260925 10:44:09.699213 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/ts-2/wal/instance
I20260925 10:44:09.699749 23952 external_mini_cluster.cc:1428] Setting key a91da5548ac04cb77622214a9e8c46ff
I20260925 10:44:09.707007 11546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:09.697195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.3:40789 (local address 127.23.100.62:44957)
0925 10:44:09.697404 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:09.697410 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:09.698746 (+ 1336us) server_negotiation.cc:407] Connection header received
0925 10:44:09.701110 (+ 2364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:09.701115 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:09.701200 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:09.701279 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:09.702475 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:09.703252 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:09.704706 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:09.705112 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:09.705202 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:09.706356 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:09.706361 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:09.706365 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:44:09.706380 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:09.706410 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:44:09.706423 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:09.706489 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:09.706707 (+ 218us) server_negotiation.cc:299] Negotiation successful
0925 10:44:09.706779 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 10:44:09.707861 11808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:09.697691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:44957 (local address 127.23.100.3:40789)
0925 10:44:09.698572 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:09.698630 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 10:44:09.700834 (+ 2204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:44:09.701537 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:44:09.701558 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:44:09.701664 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:44:09.702231 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:09.702272 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:09.703402 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:09.703406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:09.704559 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:09.704569 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:09.705293 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:09.705297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:09.705475 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:09.706057 (+ 582us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:44:09.706059 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:44:09.706096 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:44:09.706163 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:44:09.706501 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:44:09.706508 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:44:09.706513 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:44:09.706662 (+ 149us) client_negotiation.cc:240] Negotiation successful
0925 10:44:09.706928 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":248,"threads_started":1}
I20260925 10:44:09.708958 11806 heartbeater.cc:347] Connected to a master server at 127.23.100.62:44957
I20260925 10:44:09.709224 11806 heartbeater.cc:464] Registering TS with master...
I20260925 10:44:09.709892 11806 heartbeater.cc:511] Master 127.23.100.62:44957 requested a full tablet report, sending...
I20260925 10:44:09.711259 11358 ts_manager.cc:194] Registered new tserver with Master: 7995998859f44db9a61f75c7478f0f51 (127.23.100.3:34873)
I20260925 10:44:09.712054 11358 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.100.3:40789
I20260925 10:44:09.714603 23952 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:44:09.723976 11546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:09.718420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38544 (local address 127.23.100.62:44957)
0925 10:44:09.718555 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:09.718558 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:09.719081 (+ 523us) server_negotiation.cc:407] Connection header received
0925 10:44:09.719284 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:09.719288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:09.719350 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:09.719451 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:09.719935 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:09.720592 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:09.721891 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:09.722261 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:09.722428 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:09.723336 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:09.723341 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:09.723344 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:44:09.723358 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:09.723384 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:44:09.723395 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:09.723485 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:09.723625 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 10:44:09.723702 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 10:44:09.727238 11358 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38544:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:44:09.733860 11358 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:44:09.768812 11820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:09.744655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60946 (local address 127.23.100.2:46479)
0925 10:44:09.755834 (+ 11179us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:09.755847 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:09.756777 (+ 930us) server_negotiation.cc:407] Connection header received
0925 10:44:09.757204 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:09.757211 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:09.757510 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:09.757704 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:44:09.758885 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:09.760996 (+ 2111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:09.764631 (+ 3635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:09.766647 (+ 2016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:09.766783 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:09.767150 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:09.768109 (+ 959us) server_negotiation.cc:299] Negotiation successful
0925 10:44:09.768371 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10942,"thread_start_us":11181,"threads_started":1}
I20260925 10:44:09.769364 11819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:09.744400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.2:46479 (local address 127.0.0.1:60946)
0925 10:44:09.756620 (+ 12220us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:09.756664 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 10:44:09.756936 (+ 272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:44:09.758098 (+ 1162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:44:09.758108 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:44:09.758161 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:44:09.758642 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:09.758657 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:09.761240 (+ 2583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:09.761244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:09.764378 (+ 3134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:09.764399 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:09.767369 (+ 2970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:09.767378 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:09.767766 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:09.767841 (+ 75us) client_negotiation.cc:769] Sending connection context
0925 10:44:09.768841 (+ 1000us) client_negotiation.cc:240] Negotiation successful
0925 10:44:09.769048 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11980,"mutex_wait_us":50,"thread_start_us":12392,"threads_started":1}
I20260925 10:44:09.771737 11818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:09.744469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.3:34873 (local address 127.0.0.1:46336)
0925 10:44:09.757389 (+ 12920us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:09.757404 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:44:09.758078 (+ 674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:44:09.759229 (+ 1151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:44:09.759237 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:44:09.759270 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:44:09.759752 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:09.759766 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:09.764837 (+ 5071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:09.764846 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:09.769026 (+ 4180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:09.769039 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:09.770953 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:09.770959 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:09.771138 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:09.771178 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 10:44:09.771265 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 10:44:09.771388 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12461,"mutex_wait_us":1182,"thread_start_us":11774,"threads_started":1}
I20260925 10:44:09.771811 11821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:09.744866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46336 (local address 127.23.100.3:34873)
0925 10:44:09.756430 (+ 11564us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:09.756441 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:09.757490 (+ 1049us) server_negotiation.cc:407] Connection header received
0925 10:44:09.758332 (+ 842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:09.758343 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:09.758631 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:09.758829 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:44:09.761300 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:09.763332 (+ 2032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:09.769245 (+ 5913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:09.770768 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:09.770922 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:09.771196 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:09.771299 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 10:44:09.771453 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11350,"thread_start_us":11570,"threads_started":1}
I20260925 10:44:09.773499 11610 tablet_service.cc:1467] Processing CreateTablet for tablet 5ae4a61092324b3c91f6841a069cab7f (DEFAULT_TABLE table=test-table [id=2c3d95ae2a604db2bffbc3faf665c9fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:44:09.774856 11610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ae4a61092324b3c91f6841a069cab7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:44:09.774843 11740 tablet_service.cc:1467] Processing CreateTablet for tablet 5ae4a61092324b3c91f6841a069cab7f (DEFAULT_TABLE table=test-table [id=2c3d95ae2a604db2bffbc3faf665c9fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:44:09.776230 11740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ae4a61092324b3c91f6841a069cab7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:44:09.776648 11822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:09.757418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:46759 (local address 127.0.0.1:60784)
0925 10:44:09.768218 (+ 10800us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:09.768237 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:44:09.768345 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:44:09.769800 (+ 1455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:44:09.769805 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:44:09.769828 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:44:09.770129 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:09.770139 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:09.771952 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:09.771957 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:09.773818 (+ 1861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:09.773832 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:09.775941 (+ 2109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:09.775949 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:09.776112 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:09.776142 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 10:44:09.776236 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 10:44:09.776369 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10674,"thread_start_us":10376,"threads_started":1}
I20260925 10:44:09.777402 11823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:09.757499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60784 (local address 127.23.100.1:46759)
0925 10:44:09.768933 (+ 11434us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:09.768945 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:09.768980 (+ 35us) server_negotiation.cc:407] Connection header received
0925 10:44:09.769091 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:09.769101 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:09.769354 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:09.769553 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:44:09.770297 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:09.771784 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:09.774018 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:09.775723 (+ 1705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:09.775877 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:09.776081 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:09.776906 (+ 825us) server_negotiation.cc:299] Negotiation successful
0925 10:44:09.777064 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11202,"thread_start_us":11067,"threads_started":1}
I20260925 10:44:09.780679 11479 tablet_service.cc:1467] Processing CreateTablet for tablet 5ae4a61092324b3c91f6841a069cab7f (DEFAULT_TABLE table=test-table [id=2c3d95ae2a604db2bffbc3faf665c9fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:44:09.782339 11479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ae4a61092324b3c91f6841a069cab7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:44:09.785982 11824 tablet_bootstrap.cc:492] T 5ae4a61092324b3c91f6841a069cab7f P e43df83593b044a482eb86538ee7a758: Bootstrap starting.
I20260925 10:44:09.788821 11824 tablet_bootstrap.cc:654] T 5ae4a61092324b3c91f6841a069cab7f P e43df83593b044a482eb86538ee7a758: Neither blocks nor log segments found. Creating new log.
I20260925 10:44:09.789930 11825 tablet_bootstrap.cc:492] T 5ae4a61092324b3c91f6841a069cab7f P 7995998859f44db9a61f75c7478f0f51: Bootstrap starting.
I20260925 10:44:09.790160 11824 log.cc:824] T 5ae4a61092324b3c91f6841a069cab7f P e43df83593b044a482eb86538ee7a758: Log is configured to *not* fsync() on all Append() calls
I20260925 10:44:09.793207 11824 tablet_bootstrap.cc:492] T 5ae4a61092324b3c91f6841a069cab7f P e43df83593b044a482eb86538ee7a758: No bootstrap required, opened a new log
I20260925 10:44:09.793277 11825 tablet_bootstrap.cc:654] T 5ae4a61092324b3c91f6841a069cab7f P 7995998859f44db9a61f75c7478f0f51: Neither blocks nor log segments found. Creating new log.
I20260925 10:44:09.793462 11824 ts_tablet_manager.cc:1397] T 5ae4a61092324b3c91f6841a069cab7f P e43df83593b044a482eb86538ee7a758: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260925 10:44:09.794461 11825 log.cc:824] T 5ae4a61092324b3c91f6841a069cab7f P 7995998859f44db9a61f75c7478f0f51: Log is configured to *not* fsync() on all Append() calls
I20260925 10:44:09.795142 11827 tablet_bootstrap.cc:492] T 5ae4a61092324b3c91f6841a069cab7f P c124058dd1474b8c94f4c791ad5bf131: Bootstrap starting.
I20260925 10:44:09.798077 11827 tablet_bootstrap.cc:654] T 5ae4a61092324b3c91f6841a069cab7f P c124058dd1474b8c94f4c791ad5bf131: Neither blocks nor log segments found. Creating new log.
I20260925 10:44:09.798372 11825 tablet_bootstrap.cc:492] T 5ae4a61092324b3c91f6841a069cab7f P 7995998859f44db9a61f75c7478f0f51: No bootstrap required, opened a new log
I20260925 10:44:09.798619 11825 ts_tablet_manager.cc:1397] T 5ae4a61092324b3c91f6841a069cab7f P 7995998859f44db9a61f75c7478f0f51: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260925 10:44:09.799224 11827 log.cc:824] T 5ae4a61092324b3c91f6841a069cab7f P c124058dd1474b8c94f4c791ad5bf131: Log is configured to *not* fsync() on all Append() calls
I20260925 10:44:09.804277 11827 tablet_bootstrap.cc:492] T 5ae4a61092324b3c91f6841a069cab7f P c124058dd1474b8c94f4c791ad5bf131: No bootstrap required, opened a new log
I20260925 10:44:09.804556 11827 ts_tablet_manager.cc:1397] T 5ae4a61092324b3c91f6841a069cab7f P c124058dd1474b8c94f4c791ad5bf131: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260925 10:44:09.804775 11824 raft_consensus.cc:348] T 5ae4a61092324b3c91f6841a069cab7f P e43df83593b044a482eb86538ee7a758 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e43df83593b044a482eb86538ee7a758" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 46479 } } peers { permanent_uuid: "7995998859f44db9a61f75c7478f0f51" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 34873 } } peers { permanent_uuid: "c124058dd1474b8c94f4c791ad5bf131" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 46759 } }
I20260925 10:44:09.805205 11824 raft_consensus.cc:374] T 5ae4a61092324b3c91f6841a069cab7f P e43df83593b044a482eb86538ee7a758 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:44:09.805297 11824 raft_consensus.cc:729] T 5ae4a61092324b3c91f6841a069cab7f P e43df83593b044a482eb86538ee7a758 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e43df83593b044a482eb86538ee7a758, State: Initialized, Role: FOLLOWER
I20260925 10:44:09.805886 11824 consensus_queue.cc:260] T 5ae4a61092324b3c91f6841a069cab7f P e43df83593b044a482eb86538ee7a758 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e43df83593b044a482eb86538ee7a758" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 46479 } } peers { permanent_uuid: "7995998859f44db9a61f75c7478f0f51" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 34873 } } peers { permanent_uuid: "c124058dd1474b8c94f4c791ad5bf131" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 46759 } }
I20260925 10:44:09.809623 11675 heartbeater.cc:503] Master 127.23.100.62:44957 was elected leader, sending a full tablet report...
I20260925 10:44:09.810231 11824 ts_tablet_manager.cc:1428] T 5ae4a61092324b3c91f6841a069cab7f P e43df83593b044a482eb86538ee7a758: Time spent starting tablet: real 0.017s user 0.012s sys 0.005s
I20260925 10:44:09.810259 11825 raft_consensus.cc:348] T 5ae4a61092324b3c91f6841a069cab7f P 7995998859f44db9a61f75c7478f0f51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e43df83593b044a482eb86538ee7a758" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 46479 } } peers { permanent_uuid: "7995998859f44db9a61f75c7478f0f51" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 34873 } } peers { permanent_uuid: "c124058dd1474b8c94f4c791ad5bf131" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 46759 } }
I20260925 10:44:09.810871 11825 raft_consensus.cc:374] T 5ae4a61092324b3c91f6841a069cab7f P 7995998859f44db9a61f75c7478f0f51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:44:09.811007 11825 raft_consensus.cc:729] T 5ae4a61092324b3c91f6841a069cab7f P 7995998859f44db9a61f75c7478f0f51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7995998859f44db9a61f75c7478f0f51, State: Initialized, Role: FOLLOWER
I20260925 10:44:09.811805 11825 consensus_queue.cc:260] T 5ae4a61092324b3c91f6841a069cab7f P 7995998859f44db9a61f75c7478f0f51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e43df83593b044a482eb86538ee7a758" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 46479 } } peers { permanent_uuid: "7995998859f44db9a61f75c7478f0f51" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 34873 } } peers { permanent_uuid: "c124058dd1474b8c94f4c791ad5bf131" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 46759 } }
I20260925 10:44:09.816733 11806 heartbeater.cc:503] Master 127.23.100.62:44957 was elected leader, sending a full tablet report...
I20260925 10:44:09.817575 11825 ts_tablet_manager.cc:1428] T 5ae4a61092324b3c91f6841a069cab7f P 7995998859f44db9a61f75c7478f0f51: Time spent starting tablet: real 0.019s user 0.009s sys 0.009s
I20260925 10:44:09.818437 11827 raft_consensus.cc:348] T 5ae4a61092324b3c91f6841a069cab7f P c124058dd1474b8c94f4c791ad5bf131 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e43df83593b044a482eb86538ee7a758" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 46479 } } peers { permanent_uuid: "7995998859f44db9a61f75c7478f0f51" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 34873 } } peers { permanent_uuid: "c124058dd1474b8c94f4c791ad5bf131" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 46759 } }
I20260925 10:44:09.818799 11827 raft_consensus.cc:374] T 5ae4a61092324b3c91f6841a069cab7f P c124058dd1474b8c94f4c791ad5bf131 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:44:09.818943 11827 raft_consensus.cc:729] T 5ae4a61092324b3c91f6841a069cab7f P c124058dd1474b8c94f4c791ad5bf131 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c124058dd1474b8c94f4c791ad5bf131, State: Initialized, Role: FOLLOWER
I20260925 10:44:09.819522 11827 consensus_queue.cc:260] T 5ae4a61092324b3c91f6841a069cab7f P c124058dd1474b8c94f4c791ad5bf131 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e43df83593b044a482eb86538ee7a758" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 46479 } } peers { permanent_uuid: "7995998859f44db9a61f75c7478f0f51" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 34873 } } peers { permanent_uuid: "c124058dd1474b8c94f4c791ad5bf131" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 46759 } }
I20260925 10:44:09.821035 11827 ts_tablet_manager.cc:1428] T 5ae4a61092324b3c91f6841a069cab7f P c124058dd1474b8c94f4c791ad5bf131: Time spent starting tablet: real 0.016s user 0.014s sys 0.001s
I20260925 10:44:09.821089 11544 heartbeater.cc:503] Master 127.23.100.62:44957 was elected leader, sending a full tablet report...
I20260925 10:44:09.860236 11830 raft_consensus.cc:482] T 5ae4a61092324b3c91f6841a069cab7f P e43df83593b044a482eb86538ee7a758 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:44:09.860504 11830 raft_consensus.cc:504] T 5ae4a61092324b3c91f6841a069cab7f P e43df83593b044a482eb86538ee7a758 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e43df83593b044a482eb86538ee7a758" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 46479 } } peers { permanent_uuid: "7995998859f44db9a61f75c7478f0f51" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 34873 } } peers { permanent_uuid: "c124058dd1474b8c94f4c791ad5bf131" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 46759 } }
I20260925 10:44:09.862205 11830 leader_election.cc:288] T 5ae4a61092324b3c91f6841a069cab7f P e43df83593b044a482eb86538ee7a758 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7995998859f44db9a61f75c7478f0f51 (127.23.100.3:34873), c124058dd1474b8c94f4c791ad5bf131 (127.23.100.1:46759)
I20260925 10:44:09.869274 11821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:09.862622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:38979 (local address 127.23.100.3:34873)
0925 10:44:09.862846 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:09.862854 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:09.863806 (+ 952us) server_negotiation.cc:407] Connection header received
0925 10:44:09.864007 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:09.864015 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:09.864104 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:09.864187 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:44:09.864833 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:09.865553 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:09.867609 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:09.868369 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:09.868550 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:09.868619 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:09.868899 (+ 280us) server_negotiation.cc:299] Negotiation successful
0925 10:44:09.869021 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 10:44:09.869293 11823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:09.863057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.2:39455 (local address 127.23.100.1:46759)
0925 10:44:09.863288 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:09.863297 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:09.863540 (+ 243us) server_negotiation.cc:407] Connection header received
0925 10:44:09.863875 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:09.863883 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:09.863978 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:09.864054 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:44:09.864892 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:09.865552 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:09.867576 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:09.868336 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:09.868476 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:09.868550 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:09.868907 (+ 357us) server_negotiation.cc:299] Negotiation successful
0925 10:44:09.869033 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 10:44:09.869508 11833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:09.862528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:46759 (local address 127.23.100.2:39455)
0925 10:44:09.863430 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:09.863453 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 10:44:09.863701 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:44:09.864246 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:44:09.864252 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:44:09.864277 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:44:09.864731 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:09.864741 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:09.865694 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:09.865699 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:09.867356 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:09.867368 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:09.868525 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:09.868533 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:09.868714 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:09.868772 (+ 58us) client_negotiation.cc:769] Sending connection context
0925 10:44:09.868872 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 10:44:09.869018 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"mutex_wait_us":83,"thread_start_us":507,"threads_started":1}
I20260925 10:44:09.869607 11834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:09.862579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.3:34873 (local address 127.23.100.2:38979)
0925 10:44:09.863696 (+ 1117us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:09.863714 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:44:09.863856 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:44:09.864231 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:44:09.864237 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:44:09.864262 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:44:09.864666 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:09.864676 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:09.865686 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:09.865692 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:09.867358 (+ 1666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:09.867369 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:09.868586 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:09.868594 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:09.868723 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:09.868772 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 10:44:09.868874 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 10:44:09.869030 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":973,"mutex_wait_us":107,"thread_start_us":766,"threads_started":1}
I20260925 10:44:09.870573 11499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae4a61092324b3c91f6841a069cab7f" candidate_uuid: "e43df83593b044a482eb86538ee7a758" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c124058dd1474b8c94f4c791ad5bf131" is_pre_election: true
I20260925 10:44:09.870678 11760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae4a61092324b3c91f6841a069cab7f" candidate_uuid: "e43df83593b044a482eb86538ee7a758" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7995998859f44db9a61f75c7478f0f51" is_pre_election: true
I20260925 10:44:09.871124 11499 raft_consensus.cc:2454] T 5ae4a61092324b3c91f6841a069cab7f P c124058dd1474b8c94f4c791ad5bf131 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e43df83593b044a482eb86538ee7a758 in term 0.
I20260925 10:44:09.871178 11760 raft_consensus.cc:2454] T 5ae4a61092324b3c91f6841a069cab7f P 7995998859f44db9a61f75c7478f0f51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e43df83593b044a482eb86538ee7a758 in term 0.
I20260925 10:44:09.871762 11565 leader_election.cc:302] T 5ae4a61092324b3c91f6841a069cab7f P e43df83593b044a482eb86538ee7a758 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7995998859f44db9a61f75c7478f0f51, c124058dd1474b8c94f4c791ad5bf131, e43df83593b044a482eb86538ee7a758; no voters:
I20260925 10:44:09.872151 11830 raft_consensus.cc:2781] T 5ae4a61092324b3c91f6841a069cab7f P e43df83593b044a482eb86538ee7a758 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:44:09.872306 11830 raft_consensus.cc:482] T 5ae4a61092324b3c91f6841a069cab7f P e43df83593b044a482eb86538ee7a758 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:44:09.872411 11830 raft_consensus.cc:3037] T 5ae4a61092324b3c91f6841a069cab7f P e43df83593b044a482eb86538ee7a758 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:44:09.874553 11830 raft_consensus.cc:504] T 5ae4a61092324b3c91f6841a069cab7f P e43df83593b044a482eb86538ee7a758 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e43df83593b044a482eb86538ee7a758" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 46479 } } peers { permanent_uuid: "7995998859f44db9a61f75c7478f0f51" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 34873 } } peers { permanent_uuid: "c124058dd1474b8c94f4c791ad5bf131" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 46759 } }
I20260925 10:44:09.875118 11830 leader_election.cc:288] T 5ae4a61092324b3c91f6841a069cab7f P e43df83593b044a482eb86538ee7a758 [CANDIDATE]: Term 1 election: Requested vote from peers 7995998859f44db9a61f75c7478f0f51 (127.23.100.3:34873), c124058dd1474b8c94f4c791ad5bf131 (127.23.100.1:46759)
I20260925 10:44:09.875459 11760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae4a61092324b3c91f6841a069cab7f" candidate_uuid: "e43df83593b044a482eb86538ee7a758" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7995998859f44db9a61f75c7478f0f51"
I20260925 10:44:09.875499 11499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae4a61092324b3c91f6841a069cab7f" candidate_uuid: "e43df83593b044a482eb86538ee7a758" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c124058dd1474b8c94f4c791ad5bf131"
I20260925 10:44:09.875645 11760 raft_consensus.cc:3037] T 5ae4a61092324b3c91f6841a069cab7f P 7995998859f44db9a61f75c7478f0f51 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:44:09.875713 11499 raft_consensus.cc:3037] T 5ae4a61092324b3c91f6841a069cab7f P c124058dd1474b8c94f4c791ad5bf131 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:44:09.877781 11760 raft_consensus.cc:2454] T 5ae4a61092324b3c91f6841a069cab7f P 7995998859f44db9a61f75c7478f0f51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e43df83593b044a482eb86538ee7a758 in term 1.
I20260925 10:44:09.877893 11499 raft_consensus.cc:2454] T 5ae4a61092324b3c91f6841a069cab7f P c124058dd1474b8c94f4c791ad5bf131 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e43df83593b044a482eb86538ee7a758 in term 1.
I20260925 10:44:09.878204 11566 leader_election.cc:302] T 5ae4a61092324b3c91f6841a069cab7f P e43df83593b044a482eb86538ee7a758 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7995998859f44db9a61f75c7478f0f51, e43df83593b044a482eb86538ee7a758; no voters:
I20260925 10:44:09.878535 11830 raft_consensus.cc:2781] T 5ae4a61092324b3c91f6841a069cab7f P e43df83593b044a482eb86538ee7a758 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:44:09.879002 11830 raft_consensus.cc:686] T 5ae4a61092324b3c91f6841a069cab7f P e43df83593b044a482eb86538ee7a758 [term 1 LEADER]: Becoming Leader. State: Replica: e43df83593b044a482eb86538ee7a758, State: Running, Role: LEADER
I20260925 10:44:09.879499 11830 consensus_queue.cc:237] T 5ae4a61092324b3c91f6841a069cab7f P e43df83593b044a482eb86538ee7a758 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e43df83593b044a482eb86538ee7a758" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 46479 } } peers { permanent_uuid: "7995998859f44db9a61f75c7478f0f51" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 34873 } } peers { permanent_uuid: "c124058dd1474b8c94f4c791ad5bf131" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 46759 } }
I20260925 10:44:09.883703 11357 catalog_manager.cc:5495] T 5ae4a61092324b3c91f6841a069cab7f P e43df83593b044a482eb86538ee7a758 reported cstate change: term changed from 0 to 1, leader changed from <none> to e43df83593b044a482eb86538ee7a758 (127.23.100.2). New cstate: current_term: 1 leader_uuid: "e43df83593b044a482eb86538ee7a758" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e43df83593b044a482eb86538ee7a758" member_type: VOTER last_known_addr { host: "127.23.100.2" port: 46479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7995998859f44db9a61f75c7478f0f51" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 34873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c124058dd1474b8c94f4c791ad5bf131" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 46759 } health_report { overall_health: UNKNOWN } } }
I20260925 10:44:09.899013 11820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:09.893819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60956 (local address 127.23.100.2:46479)
0925 10:44:09.894018 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:09.894025 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:09.894116 (+ 91us) server_negotiation.cc:407] Connection header received
0925 10:44:09.894360 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:09.894364 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:09.894433 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:09.894595 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:44:09.894944 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:09.895656 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:09.896992 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:09.897385 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:09.897515 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:09.897959 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:44:09.898464 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:44:09.898614 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:09.898666 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 10:44:09.898736 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 10:44:09.906594 11499 raft_consensus.cc:1261] T 5ae4a61092324b3c91f6841a069cab7f P c124058dd1474b8c94f4c791ad5bf131 [term 1 FOLLOWER]: Refusing update from remote peer e43df83593b044a482eb86538ee7a758: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:44:09.906599 11760 raft_consensus.cc:1261] T 5ae4a61092324b3c91f6841a069cab7f P 7995998859f44db9a61f75c7478f0f51 [term 1 FOLLOWER]: Refusing update from remote peer e43df83593b044a482eb86538ee7a758: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:44:09.907473 11835 consensus_queue.cc:1035] T 5ae4a61092324b3c91f6841a069cab7f P e43df83593b044a482eb86538ee7a758 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c124058dd1474b8c94f4c791ad5bf131" member_type: VOTER last_known_addr { host: "127.23.100.1" port: 46759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:44:09.907817 11830 consensus_queue.cc:1035] T 5ae4a61092324b3c91f6841a069cab7f P e43df83593b044a482eb86538ee7a758 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7995998859f44db9a61f75c7478f0f51" member_type: VOTER last_known_addr { host: "127.23.100.3" port: 34873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:44:09.929348 11841 mvcc.cc:205] Tried to move back new op lower bound from 7333204172408033280 to 7333204172310183936. Current Snapshot: MvccSnapshot[applied={T|T < 7333204172408033280}]
I20260925 10:44:09.931689 11357 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:44:09.932008 11357 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:44:09.937129 11357 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5a557e9b349c4a53a4a656488694f36e: Sending DeleteTablet for 3 replicas of tablet 5ae4a61092324b3c91f6841a069cab7f
I20260925 10:44:09.938104 11610 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ae4a61092324b3c91f6841a069cab7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:44:09 UTC) from {username='slave'} at 127.0.0.1:60946
I20260925 10:44:09.938102 11740 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ae4a61092324b3c91f6841a069cab7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:44:09 UTC) from {username='slave'} at 127.0.0.1:46336
I20260925 10:44:09.938719 11479 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ae4a61092324b3c91f6841a069cab7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:44:09 UTC) from {username='slave'} at 127.0.0.1:60784
I20260925 10:44:09.939193 11739 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ae4a61092324b3c91f6841a069cab7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:44:09 UTC) from {username='slave'} at 127.0.0.1:46336
I20260925 10:44:09.940936 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 11417
I20260925 10:44:09.949827 11848 tablet_replica.cc:330] T 5ae4a61092324b3c91f6841a069cab7f P e43df83593b044a482eb86538ee7a758: stopping tablet replica
I20260925 10:44:09.950338 11848 raft_consensus.cc:2229] T 5ae4a61092324b3c91f6841a069cab7f P e43df83593b044a482eb86538ee7a758 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:44:09.951110 11848 raft_consensus.cc:2258] T 5ae4a61092324b3c91f6841a069cab7f P e43df83593b044a482eb86538ee7a758 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:44:09.955650 11848 ts_tablet_manager.cc:1905] T 5ae4a61092324b3c91f6841a069cab7f P e43df83593b044a482eb86538ee7a758: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:44:09.959913 11849 tablet_replica.cc:330] T 5ae4a61092324b3c91f6841a069cab7f P c124058dd1474b8c94f4c791ad5bf131: stopping tablet replica
I20260925 10:44:09.960327 11849 raft_consensus.cc:2229] T 5ae4a61092324b3c91f6841a069cab7f P c124058dd1474b8c94f4c791ad5bf131 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:44:09.960807 11849 raft_consensus.cc:2258] T 5ae4a61092324b3c91f6841a069cab7f P c124058dd1474b8c94f4c791ad5bf131 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:44:09.965677 11823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:09.942696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60796 (local address 127.23.100.1:46759)
0925 10:44:09.942936 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:09.942945 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:09.955089 (+ 12144us) server_negotiation.cc:407] Connection header received
0925 10:44:09.955536 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:09.955541 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:09.955674 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:09.955756 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:09.956435 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:09.957607 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:09.961170 (+ 3563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:09.961865 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:09.961996 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:09.963910 (+ 1914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:09.963921 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:09.963938 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:44:09.964681 (+ 743us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:09.964737 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:44:09.964764 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:09.964871 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:09.965089 (+ 218us) server_negotiation.cc:299] Negotiation successful
0925 10:44:09.965199 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 10:44:09.965364 11849 ts_tablet_manager.cc:1905] T 5ae4a61092324b3c91f6841a069cab7f P c124058dd1474b8c94f4c791ad5bf131: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:44:09.966687 11539 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:44:09.967414 11847 tablet_replica.cc:330] T 5ae4a61092324b3c91f6841a069cab7f P 7995998859f44db9a61f75c7478f0f51: stopping tablet replica
I20260925 10:44:09.967787 11847 raft_consensus.cc:2229] T 5ae4a61092324b3c91f6841a069cab7f P 7995998859f44db9a61f75c7478f0f51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:44:09.968127 11848 ts_tablet_manager.cc:1918] T 5ae4a61092324b3c91f6841a069cab7f P e43df83593b044a482eb86538ee7a758: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:44:09.968268 11847 raft_consensus.cc:2258] T 5ae4a61092324b3c91f6841a069cab7f P 7995998859f44db9a61f75c7478f0f51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:44:09.968288 11848 log.cc:1196] T 5ae4a61092324b3c91f6841a069cab7f P e43df83593b044a482eb86538ee7a758: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/ts-1/wal/wals/5ae4a61092324b3c91f6841a069cab7f
I20260925 10:44:09.969264 11848 ts_tablet_manager.cc:1939] T 5ae4a61092324b3c91f6841a069cab7f P e43df83593b044a482eb86538ee7a758: Deleting consensus metadata
I20260925 10:44:09.969914 11847 ts_tablet_manager.cc:1905] T 5ae4a61092324b3c91f6841a069cab7f P 7995998859f44db9a61f75c7478f0f51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:44:09.971045 11344 catalog_manager.cc:4841] TS e43df83593b044a482eb86538ee7a758 (127.23.100.2:46479): tablet 5ae4a61092324b3c91f6841a069cab7f (table test-table [id=2c3d95ae2a604db2bffbc3faf665c9fe]) successfully deleted
I20260925 10:44:09.974988 11847 ts_tablet_manager.cc:1918] T 5ae4a61092324b3c91f6841a069cab7f P 7995998859f44db9a61f75c7478f0f51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:44:09.975199 11847 log.cc:1196] T 5ae4a61092324b3c91f6841a069cab7f P 7995998859f44db9a61f75c7478f0f51: Deleting WAL directory at /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790332740062128-23952-0/minicluster-data/ts-2/wal/wals/5ae4a61092324b3c91f6841a069cab7f
I20260925 10:44:09.975993 11847 ts_tablet_manager.cc:1939] T 5ae4a61092324b3c91f6841a069cab7f P 7995998859f44db9a61f75c7478f0f51: Deleting consensus metadata
I20260925 10:44:09.978191 11345 catalog_manager.cc:4841] TS 7995998859f44db9a61f75c7478f0f51 (127.23.100.3:34873): tablet 5ae4a61092324b3c91f6841a069cab7f (table test-table [id=2c3d95ae2a604db2bffbc3faf665c9fe]) successfully deleted
W20260925 10:44:09.979131 11345 catalog_manager.cc:4808] TS 7995998859f44db9a61f75c7478f0f51 (127.23.100.3:34873): delete failed for tablet 5ae4a61092324b3c91f6841a069cab7f because the tablet was not found. No further retry: Not found: Tablet not found: 5ae4a61092324b3c91f6841a069cab7f
I20260925 10:44:10.063695 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 11417
W20260925 10:44:10.081385 11344 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.100.1:46759 (error 108)
I20260925 10:44:10.081597 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 11548
I20260925 10:44:10.083796 11822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:10.083293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:46759 (local address 127.0.0.1:60804)
0925 10:44:10.083554 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:10.083669 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.1:46759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
W20260925 10:44:10.084080 11344 catalog_manager.cc:4584] TS c124058dd1474b8c94f4c791ad5bf131 (127.23.100.1:46759): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5ae4a61092324b3c91f6841a069cab7f: Network error: Client connection negotiation failed: client connection to 127.23.100.1:46759: connect: Connection refused (error 111)
I20260925 10:44:10.089859 11820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:10.082890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60964 (local address 127.23.100.2:46479)
0925 10:44:10.083075 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:10.083080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:10.083098 (+ 18us) server_negotiation.cc:407] Connection header received
0925 10:44:10.083163 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:10.083171 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:10.083281 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:10.083389 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:10.083903 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:10.084844 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:10.086784 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:10.087200 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:10.087297 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:10.088449 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:10.088459 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:10.088473 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:44:10.089075 (+ 602us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:10.089126 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:44:10.089145 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:10.089225 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:10.089403 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 10:44:10.089491 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 10:44:10.090539 11670 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:44:10.137737 11822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:10.137205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:46759 (local address 127.0.0.1:60812)
0925 10:44:10.137426 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:10.137540 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.1:46759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260925 10:44:10.138803 11822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:10.138426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:46759 (local address 127.0.0.1:60820)
0925 10:44:10.138578 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:10.138661 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.1:46759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 10:44:10.199460 11822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:10.198839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:46759 (local address 127.0.0.1:60822)
0925 10:44:10.199143 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:10.199242 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.1:46759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185}
I20260925 10:44:10.200924 11822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:10.200167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:46759 (local address 127.0.0.1:60832)
0925 10:44:10.200664 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:10.200771 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.1:46759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342}
I20260925 10:44:10.202026 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 11548
I20260925 10:44:10.221536 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 11678
I20260925 10:44:10.228087 11821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:10.222239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46348 (local address 127.23.100.3:34873)
0925 10:44:10.222455 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:10.222460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:10.222540 (+ 80us) server_negotiation.cc:407] Connection header received
0925 10:44:10.222691 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:10.222695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:10.222775 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:10.222855 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:10.223226 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:10.223928 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:10.225090 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:10.225450 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:10.225539 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:10.226774 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:10.226781 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:10.226795 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:44:10.227399 (+ 604us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:10.227444 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:44:10.227464 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:10.227545 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:10.227701 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 10:44:10.227780 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 10:44:10.228718 11801 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:44:10.281574 11822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:10.281070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:46759 (local address 127.0.0.1:60842)
0925 10:44:10.281280 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:10.281398 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.1:46759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260925 10:44:10.282771 11822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:10.282318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.1:46759 (local address 127.0.0.1:60850)
0925 10:44:10.282546 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:10.282630 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.1:46759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260925 10:44:10.319336 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 11678
I20260925 10:44:10.335516 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 11327
I20260925 10:44:10.336603 11388 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:44:10.442772 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 11327
2026-09-25T10:44:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2665 ms)
[----------] 6 tests from EncryptionPolicyTest (21439 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790332740062128-23952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:44:10 dist-test-slave-dxdd krb5kdc[11858](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:44:10 dist-test-slave-dxdd krb5kdc[11858](info): set up 2 sockets
Sep 25 10:44:10 dist-test-slave-dxdd krb5kdc[11858](info): commencing operation
krb5kdc: starting...
W20260925 10:44:11.539041 23952 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Sep 25 10:44:11 dist-test-slave-dxdd krb5kdc[11858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:44:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:44:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790332740062128-23952-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790332740062128-23952-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:44:11.773811 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45061
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790332740062128-23952-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:44:12.091621 11871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:44:12.091938 11871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:44:12.092132 11871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:44:12.099231 11871 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:44:12.099371 11871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:44:12.099486 11871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:44:12.099608 11871 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:44:12.099684 11871 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:44:12.109078 11871 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790332740062128-23952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45061
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790332740062128-23952-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:44:12.110319 11871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:44:12.111563 11871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:44:12.122956 11877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:12.123027 11876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:12.123054 11879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:44:12.124487 11871 server_base.cc:1034] running on GCE node
I20260925 10:44:12.125226 11871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:44:12.126560 11871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:44:12.127787 11871 hybrid_clock.cc:648] HybridClock initialized: now 1790333052127751 us; error 58 us; skew 500 ppm
I20260925 10:44:12.128132 11871 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:44:12 dist-test-slave-dxdd krb5kdc[11858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:44:12.143913 11871 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:44:12.145642 11871 webserver.cc:460] Webserver started at http://127.0.0.1:36397/ using document root <none> and password file <none>
I20260925 10:44:12.146421 11871 fs_manager.cc:359] Metadata directory not provided
I20260925 10:44:12.146574 11871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:44:12.146878 11871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:44:12.150132 11871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790332740062128-23952-0/minicluster-data/master-0/data/instance:
uuid: "c02aa01ba3b348b4adc90fc17d7db02a"
format_stamp: "Formatted at 2026-09-25 10:44:12 on dist-test-slave-dxdd"
server_key: "e6a344ca05ba961c3bbc0f67b0800fa2"
server_key_iv: "0116d8a7ae937a4a5ef132f9b8db1e20"
server_key_version: "encryptionkey@0"
I20260925 10:44:12.150995 11871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790332740062128-23952-0/minicluster-data/master-0/wal/instance:
uuid: "c02aa01ba3b348b4adc90fc17d7db02a"
format_stamp: "Formatted at 2026-09-25 10:44:12 on dist-test-slave-dxdd"
server_key: "e6a344ca05ba961c3bbc0f67b0800fa2"
server_key_iv: "0116d8a7ae937a4a5ef132f9b8db1e20"
server_key_version: "encryptionkey@0"
I20260925 10:44:12.156054 11871 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 10:44:12.159502 11886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:44:12.160758 11871 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 10:44:12.160941 11871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790332740062128-23952-0/minicluster-data/master-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790332740062128-23952-0/minicluster-data/master-0/wal
uuid: "c02aa01ba3b348b4adc90fc17d7db02a"
format_stamp: "Formatted at 2026-09-25 10:44:12 on dist-test-slave-dxdd"
server_key: "e6a344ca05ba961c3bbc0f67b0800fa2"
server_key_iv: "0116d8a7ae937a4a5ef132f9b8db1e20"
server_key_version: "encryptionkey@0"
I20260925 10:44:12.161139 11871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790332740062128-23952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790332740062128-23952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790332740062128-23952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:44:12.181579 11871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:44:12.185676 11871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:44:12.185964 11871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:44:12.204252 11871 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45061
I20260925 10:44:12.204283 11937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45061 every 8 connection(s)
I20260925 10:44:12.205524 11871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
I20260925 10:44:12.206188 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 11871
I20260925 10:44:12.206468 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790332740062128-23952-0/minicluster-data/master-0/wal/instance
I20260925 10:44:12.207098 23952 external_mini_cluster.cc:1428] Setting key cc896ee02f90bc361196254d9aaa2588
I20260925 10:44:12.210165 11938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 10:44:12 dist-test-slave-dxdd krb5kdc[11858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 10:44:12.220522 11938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c02aa01ba3b348b4adc90fc17d7db02a: Bootstrap starting.
I20260925 10:44:12.223862 11938 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c02aa01ba3b348b4adc90fc17d7db02a: Neither blocks nor log segments found. Creating new log.
I20260925 10:44:12.224898 11938 log.cc:824] T 00000000000000000000000000000000 P c02aa01ba3b348b4adc90fc17d7db02a: Log is configured to *not* fsync() on all Append() calls
I20260925 10:44:12.228266 11938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c02aa01ba3b348b4adc90fc17d7db02a: No bootstrap required, opened a new log
I20260925 10:44:12.229327 11941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:12.209130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50993 (local address 127.0.0.1:45061)
0925 10:44:12.209891 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:12.209903 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:12.209941 (+ 38us) server_negotiation.cc:407] Connection header received
0925 10:44:12.211682 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:12.211705 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:12.212064 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:12.212460 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:12.213888 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:12.215253 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:12.216509 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:12.217142 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:12.217345 (+ 203us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:12.221484 (+ 4139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:12.221492 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:12.221505 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:44:12.221551 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:12.224645 (+ 3094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:12.225527 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:12.225535 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:12.225543 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:12.225722 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:12.226279 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:12.226287 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:12.226293 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:12.226990 (+ 697us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:12.227307 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:12.227699 (+ 392us) server_negotiation.cc:299] Negotiation successful
0925 10:44:12.228022 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":372,"threads_started":1}
I20260925 10:44:12.233276 23952 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:44:12.236758 11938 raft_consensus.cc:348] T 00000000000000000000000000000000 P c02aa01ba3b348b4adc90fc17d7db02a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c02aa01ba3b348b4adc90fc17d7db02a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45061 } }
I20260925 10:44:12.237071 11938 raft_consensus.cc:374] T 00000000000000000000000000000000 P c02aa01ba3b348b4adc90fc17d7db02a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:44:12.237193 11938 raft_consensus.cc:729] T 00000000000000000000000000000000 P c02aa01ba3b348b4adc90fc17d7db02a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c02aa01ba3b348b4adc90fc17d7db02a, State: Initialized, Role: FOLLOWER
I20260925 10:44:12.237833 11938 consensus_queue.cc:260] T 00000000000000000000000000000000 P c02aa01ba3b348b4adc90fc17d7db02a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c02aa01ba3b348b4adc90fc17d7db02a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45061 } }
I20260925 10:44:12.238142 11938 raft_consensus.cc:388] T 00000000000000000000000000000000 P c02aa01ba3b348b4adc90fc17d7db02a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:44:12.238292 11938 raft_consensus.cc:482] T 00000000000000000000000000000000 P c02aa01ba3b348b4adc90fc17d7db02a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:44:12.238516 11938 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c02aa01ba3b348b4adc90fc17d7db02a [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:44:12.241055 11938 raft_consensus.cc:504] T 00000000000000000000000000000000 P c02aa01ba3b348b4adc90fc17d7db02a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c02aa01ba3b348b4adc90fc17d7db02a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45061 } }
I20260925 10:44:12.241593 11938 leader_election.cc:302] T 00000000000000000000000000000000 P c02aa01ba3b348b4adc90fc17d7db02a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c02aa01ba3b348b4adc90fc17d7db02a; no voters:
I20260925 10:44:12.242358 11938 leader_election.cc:288] T 00000000000000000000000000000000 P c02aa01ba3b348b4adc90fc17d7db02a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:44:12.242538 11948 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c02aa01ba3b348b4adc90fc17d7db02a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:44:12.245040 11938 sys_catalog.cc:564] T 00000000000000000000000000000000 P c02aa01ba3b348b4adc90fc17d7db02a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:44:12.246649 11948 raft_consensus.cc:686] T 00000000000000000000000000000000 P c02aa01ba3b348b4adc90fc17d7db02a [term 1 LEADER]: Becoming Leader. State: Replica: c02aa01ba3b348b4adc90fc17d7db02a, State: Running, Role: LEADER
I20260925 10:44:12.247380 11948 consensus_queue.cc:237] T 00000000000000000000000000000000 P c02aa01ba3b348b4adc90fc17d7db02a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c02aa01ba3b348b4adc90fc17d7db02a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45061 } }
I20260925 10:44:12.250456 11950 sys_catalog.cc:455] T 00000000000000000000000000000000 P c02aa01ba3b348b4adc90fc17d7db02a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c02aa01ba3b348b4adc90fc17d7db02a. Latest consensus state: current_term: 1 leader_uuid: "c02aa01ba3b348b4adc90fc17d7db02a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c02aa01ba3b348b4adc90fc17d7db02a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45061 } } }
I20260925 10:44:12.250864 11950 sys_catalog.cc:458] T 00000000000000000000000000000000 P c02aa01ba3b348b4adc90fc17d7db02a [sys.catalog]: This master's current role is: LEADER
I20260925 10:44:12.251365 11949 sys_catalog.cc:455] T 00000000000000000000000000000000 P c02aa01ba3b348b4adc90fc17d7db02a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c02aa01ba3b348b4adc90fc17d7db02a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c02aa01ba3b348b4adc90fc17d7db02a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45061 } } }
I20260925 10:44:12.251695 11949 sys_catalog.cc:458] T 00000000000000000000000000000000 P c02aa01ba3b348b4adc90fc17d7db02a [sys.catalog]: This master's current role is: LEADER
I20260925 10:44:12.251976 11954 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:44:12.253733 11941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:12.237901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46069 (local address 127.0.0.1:45061)
0925 10:44:12.238037 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:12.238041 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:12.238657 (+ 616us) server_negotiation.cc:407] Connection header received
0925 10:44:12.238792 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:12.238796 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:12.238871 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:12.239004 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:12.240359 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:12.241423 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:12.243260 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:12.243893 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:12.243995 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:12.246653 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:12.246660 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:12.246669 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:44:12.246710 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:12.249865 (+ 3155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:12.250928 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:12.250934 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:12.250937 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:12.251153 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:12.251799 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:12.251803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:12.251807 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:12.252223 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:12.252510 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:12.253210 (+ 700us) server_negotiation.cc:299] Negotiation successful
0925 10:44:12.253373 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 10:44:12.267788 11954 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:44:12.277385 11954 catalog_manager.cc:1347] Generated new cluster ID: 90364442c9cc459db6b72295e9382479
I20260925 10:44:12.277482 11954 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:44:12.295677 11954 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:44:12.296751 11954 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:44:12.311194 11954 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c02aa01ba3b348b4adc90fc17d7db02a: Generated new TSK 0
I20260925 10:44:12.311941 11954 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:44:12.326568 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 11871
I20260925 10:44:12.327742 11933 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:44:12.419154 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 11871
W20260925 10:44:12.436348 11856 connection.cc:535] client connection to 127.0.0.1:45061 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45061): Connection reset by peer (error 104)
2026-09-25T10:44:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1972 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790332740062128-23952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:44:12 dist-test-slave-dxdd krb5kdc[11970](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:44:12 dist-test-slave-dxdd krb5kdc[11970](info): set up 2 sockets
Sep 25 10:44:12 dist-test-slave-dxdd krb5kdc[11970](info): commencing operation
krb5kdc: starting...
W20260925 10:44:13.514858 23952 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Sep 25 10:44:13 dist-test-slave-dxdd krb5kdc[11970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:44:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:44:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790332740062128-23952-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790332740062128-23952-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:44:13.752986 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33387
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790332740062128-23952-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:44:14.070849 11983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:44:14.071120 11983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:44:14.071208 11983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:44:14.078456 11983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:44:14.078564 11983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:44:14.078610 11983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:44:14.078677 11983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:44:14.078722 11983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:44:14.089025 11983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790332740062128-23952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33387
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790332740062128-23952-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:44:14.090528 11983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:44:14.091847 11983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:44:14.103662 11991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:14.103744 11988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:14.103669 11989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:44:14.104581 11983 server_base.cc:1034] running on GCE node
I20260925 10:44:14.105280 11983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:44:14.106770 11983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:44:14.107997 11983 hybrid_clock.cc:648] HybridClock initialized: now 1790333054107959 us; error 50 us; skew 500 ppm
I20260925 10:44:14.108341 11983 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:44:14 dist-test-slave-dxdd krb5kdc[11970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333054, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:44:14.111406 11983 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:44:14.112867 11983 webserver.cc:460] Webserver started at http://127.0.0.1:32777/ using document root <none> and password file <none>
I20260925 10:44:14.113627 11983 fs_manager.cc:359] Metadata directory not provided
I20260925 10:44:14.113782 11983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:44:14.114077 11983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:44:14.116956 11983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790332740062128-23952-0/minicluster-data/master-0/data/instance:
uuid: "60c35b80825f4c29aeb7e07c4802ea6d"
format_stamp: "Formatted at 2026-09-25 10:44:14 on dist-test-slave-dxdd"
server_key: "49c470d74a3d06f70d31acb62051a15c"
server_key_iv: "bae3b85edce417e13bf032b672dfe27a"
server_key_version: "encryptionkey@0"
I20260925 10:44:14.117710 11983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790332740062128-23952-0/minicluster-data/master-0/wal/instance:
uuid: "60c35b80825f4c29aeb7e07c4802ea6d"
format_stamp: "Formatted at 2026-09-25 10:44:14 on dist-test-slave-dxdd"
server_key: "49c470d74a3d06f70d31acb62051a15c"
server_key_iv: "bae3b85edce417e13bf032b672dfe27a"
server_key_version: "encryptionkey@0"
I20260925 10:44:14.122602 11983 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260925 10:44:14.125854 11998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:44:14.127135 11983 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:44:14.127316 11983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790332740062128-23952-0/minicluster-data/master-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790332740062128-23952-0/minicluster-data/master-0/wal
uuid: "60c35b80825f4c29aeb7e07c4802ea6d"
format_stamp: "Formatted at 2026-09-25 10:44:14 on dist-test-slave-dxdd"
server_key: "49c470d74a3d06f70d31acb62051a15c"
server_key_iv: "bae3b85edce417e13bf032b672dfe27a"
server_key_version: "encryptionkey@0"
I20260925 10:44:14.127501 11983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790332740062128-23952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790332740062128-23952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790332740062128-23952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:44:14.161911 11983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:44:14.165936 11983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:44:14.166222 11983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:44:14.184100 11983 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33387
I20260925 10:44:14.184135 12049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33387 every 8 connection(s)
I20260925 10:44:14.185465 11983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
I20260925 10:44:14.186589 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 11983
I20260925 10:44:14.186772 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790332740062128-23952-0/minicluster-data/master-0/wal/instance
I20260925 10:44:14.187207 23952 external_mini_cluster.cc:1428] Setting key 63ee5afd60172cdd271b869c0a7b8b76
I20260925 10:44:14.189855 12050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:44:14.201879 12053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:14.189077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42317 (local address 127.0.0.1:33387)
0925 10:44:14.189905 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:14.189922 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:14.189970 (+ 48us) server_negotiation.cc:407] Connection header received
0925 10:44:14.192193 (+ 2223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:14.192222 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:14.192705 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:14.193119 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:14.193582 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:14.195541 (+ 1959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:14.196830 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:14.197794 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:14.197956 (+ 162us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:14.199216 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:14.199227 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:14.199243 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:44:14.199967 (+ 724us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:14.200055 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:44:14.200088 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:14.200250 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:14.200483 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 10:44:14.200656 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":334,"threads_started":1}
I20260925 10:44:14.203341 12050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60c35b80825f4c29aeb7e07c4802ea6d: Bootstrap starting.
I20260925 10:44:14.205493 23952 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:44:14.206561 12050 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60c35b80825f4c29aeb7e07c4802ea6d: Neither blocks nor log segments found. Creating new log.
I20260925 10:44:14.207507 12050 log.cc:824] T 00000000000000000000000000000000 P 60c35b80825f4c29aeb7e07c4802ea6d: Log is configured to *not* fsync() on all Append() calls
I20260925 10:44:14.210256 12050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60c35b80825f4c29aeb7e07c4802ea6d: No bootstrap required, opened a new log
I20260925 10:44:14.216403 12053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:14.209989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47727 (local address 127.0.0.1:33387)
0925 10:44:14.210142 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:14.210147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:14.210553 (+ 406us) server_negotiation.cc:407] Connection header received
0925 10:44:14.210713 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:14.210718 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:14.210812 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:14.210918 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:14.211317 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:14.212416 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:14.213871 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:14.214564 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:14.214695 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:14.215698 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:14.215704 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:14.215709 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:44:14.215727 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:14.215782 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:44:14.215800 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:14.215914 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:14.216030 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 10:44:14.216080 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 10:44:14.219419 12050 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60c35b80825f4c29aeb7e07c4802ea6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60c35b80825f4c29aeb7e07c4802ea6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33387 } }
I20260925 10:44:14.219730 12050 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60c35b80825f4c29aeb7e07c4802ea6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:44:14.219801 12050 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60c35b80825f4c29aeb7e07c4802ea6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60c35b80825f4c29aeb7e07c4802ea6d, State: Initialized, Role: FOLLOWER
I20260925 10:44:14.220255 12050 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60c35b80825f4c29aeb7e07c4802ea6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60c35b80825f4c29aeb7e07c4802ea6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33387 } }
I20260925 10:44:14.220445 12050 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60c35b80825f4c29aeb7e07c4802ea6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:44:14.220535 12050 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60c35b80825f4c29aeb7e07c4802ea6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:44:14.220657 12050 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60c35b80825f4c29aeb7e07c4802ea6d [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:44:14.222903 12050 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60c35b80825f4c29aeb7e07c4802ea6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60c35b80825f4c29aeb7e07c4802ea6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33387 } }
I20260925 10:44:14.223417 12050 leader_election.cc:302] T 00000000000000000000000000000000 P 60c35b80825f4c29aeb7e07c4802ea6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60c35b80825f4c29aeb7e07c4802ea6d; no voters:
I20260925 10:44:14.223865 12050 leader_election.cc:288] T 00000000000000000000000000000000 P 60c35b80825f4c29aeb7e07c4802ea6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:44:14.224028 12060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60c35b80825f4c29aeb7e07c4802ea6d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:44:14.224828 12060 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60c35b80825f4c29aeb7e07c4802ea6d [term 1 LEADER]: Becoming Leader. State: Replica: 60c35b80825f4c29aeb7e07c4802ea6d, State: Running, Role: LEADER
I20260925 10:44:14.225337 12060 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60c35b80825f4c29aeb7e07c4802ea6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60c35b80825f4c29aeb7e07c4802ea6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33387 } }
I20260925 10:44:14.225816 12050 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60c35b80825f4c29aeb7e07c4802ea6d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:44:14.228242 12061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60c35b80825f4c29aeb7e07c4802ea6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60c35b80825f4c29aeb7e07c4802ea6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60c35b80825f4c29aeb7e07c4802ea6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33387 } } }
I20260925 10:44:14.228305 12062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60c35b80825f4c29aeb7e07c4802ea6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60c35b80825f4c29aeb7e07c4802ea6d. Latest consensus state: current_term: 1 leader_uuid: "60c35b80825f4c29aeb7e07c4802ea6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60c35b80825f4c29aeb7e07c4802ea6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33387 } } }
I20260925 10:44:14.228608 12061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60c35b80825f4c29aeb7e07c4802ea6d [sys.catalog]: This master's current role is: LEADER
I20260925 10:44:14.228677 12062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60c35b80825f4c29aeb7e07c4802ea6d [sys.catalog]: This master's current role is: LEADER
I20260925 10:44:14.230504 12066 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:44:14.235431 12066 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:44:14.243000 12066 catalog_manager.cc:1347] Generated new cluster ID: 60cda54d833244da9d8c67c3c47e9f58
I20260925 10:44:14.243108 12066 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:44:14.257984 12066 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:44:14.259723 12066 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:44:14.278118 12066 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60c35b80825f4c29aeb7e07c4802ea6d: Generated new TSK 0
I20260925 10:44:14.279229 12066 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:44:14.291074 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 11983
W20260925 10:44:14.292205 12045 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42317
W20260925 10:44:14.292904 23952 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 11983: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:44:14.293102 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 11983
W20260925 10:44:14.309988 11968 connection.cc:535] client connection to 127.0.0.1:33387 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33387): Connection reset by peer (error 104)
2026-09-25T10:44:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1872 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790332740062128-23952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:44:14 dist-test-slave-dxdd krb5kdc[12081](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:44:14 dist-test-slave-dxdd krb5kdc[12081](info): set up 2 sockets
Sep 25 10:44:14 dist-test-slave-dxdd krb5kdc[12081](info): commencing operation
krb5kdc: starting...
W20260925 10:44:15.378715 23952 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Sep 25 10:44:15 dist-test-slave-dxdd krb5kdc[12081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:44:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:44:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790332740062128-23952-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790332740062128-23952-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:44:15.617372 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43055
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790332740062128-23952-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:44:15.954172 12094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:44:15.954535 12094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:44:15.954715 12094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:44:15.962345 12094 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:44:15.962482 12094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:44:15.962572 12094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:44:15.962656 12094 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:44:15.962733 12094 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:44:15.972644 12094 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790332740062128-23952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43055
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790332740062128-23952-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:44:15.973831 12094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:44:15.975199 12094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:44:15.986639 12099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:15.986589 12100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:15.986688 12102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:44:15.987564 12094 server_base.cc:1034] running on GCE node
I20260925 10:44:15.988358 12094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:44:15.989698 12094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:44:15.990931 12094 hybrid_clock.cc:648] HybridClock initialized: now 1790333055990881 us; error 63 us; skew 500 ppm
I20260925 10:44:15.991279 12094 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:44:15 dist-test-slave-dxdd krb5kdc[12081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:44:15.994244 12094 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:44:15.995678 12094 webserver.cc:460] Webserver started at http://127.0.0.1:42631/ using document root <none> and password file <none>
I20260925 10:44:15.996305 12094 fs_manager.cc:359] Metadata directory not provided
I20260925 10:44:15.996402 12094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:44:15.996639 12094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:44:15.999529 12094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790332740062128-23952-0/minicluster-data/master-0/data/instance:
uuid: "7b3249f1ad3b4dfb8051a01e9331aaaa"
format_stamp: "Formatted at 2026-09-25 10:44:15 on dist-test-slave-dxdd"
server_key: "42fcf7ff14330e45ebd4406d9d5a715f"
server_key_iv: "17560af5c61231affba77adfc0e3306a"
server_key_version: "encryptionkey@0"
I20260925 10:44:16.000232 12094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790332740062128-23952-0/minicluster-data/master-0/wal/instance:
uuid: "7b3249f1ad3b4dfb8051a01e9331aaaa"
format_stamp: "Formatted at 2026-09-25 10:44:15 on dist-test-slave-dxdd"
server_key: "42fcf7ff14330e45ebd4406d9d5a715f"
server_key_iv: "17560af5c61231affba77adfc0e3306a"
server_key_version: "encryptionkey@0"
I20260925 10:44:16.005231 12094 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 10:44:16.008591 12109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:44:16.009841 12094 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:44:16.010010 12094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790332740062128-23952-0/minicluster-data/master-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790332740062128-23952-0/minicluster-data/master-0/wal
uuid: "7b3249f1ad3b4dfb8051a01e9331aaaa"
format_stamp: "Formatted at 2026-09-25 10:44:15 on dist-test-slave-dxdd"
server_key: "42fcf7ff14330e45ebd4406d9d5a715f"
server_key_iv: "17560af5c61231affba77adfc0e3306a"
server_key_version: "encryptionkey@0"
I20260925 10:44:16.010213 12094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790332740062128-23952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790332740062128-23952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790332740062128-23952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:44:16.015758 12094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:44:16.019557 12094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:44:16.019852 12094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:44:16.038255 12094 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43055
I20260925 10:44:16.038347 12160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43055 every 8 connection(s)
I20260925 10:44:16.039677 12094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
I20260925 10:44:16.041328 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 12094
I20260925 10:44:16.041494 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790332740062128-23952-0/minicluster-data/master-0/wal/instance
I20260925 10:44:16.041949 23952 external_mini_cluster.cc:1428] Setting key 68d6ddd53e19246fc1fe6a47b7705b75
I20260925 10:44:16.043774 12161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:44:16.049537 12164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:16.043679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54211 (local address 127.0.0.1:43055)
0925 10:44:16.044576 (+ 897us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:16.044588 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:16.044630 (+ 42us) server_negotiation.cc:407] Connection header received
0925 10:44:16.046219 (+ 1589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:16.046238 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:16.046609 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:16.046962 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:16.048073 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:16.048079 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:16.048090 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:44:16.048114 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:16.048183 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:44:16.048201 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:16.048356 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:16.048492 (+ 136us) server_negotiation.cc:299] Negotiation successful
0925 10:44:16.048677 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":466,"threads_started":1}
I20260925 10:44:16.053066 23952 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:44:16.056841 12161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b3249f1ad3b4dfb8051a01e9331aaaa: Bootstrap starting.
I20260925 10:44:16.060534 12161 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b3249f1ad3b4dfb8051a01e9331aaaa: Neither blocks nor log segments found. Creating new log.
I20260925 10:44:16.060787 12164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:16.058491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33759 (local address 127.0.0.1:43055)
0925 10:44:16.058677 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:16.058684 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:16.058896 (+ 212us) server_negotiation.cc:407] Connection header received
0925 10:44:16.059060 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:16.059070 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:16.059128 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:16.059238 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:16.060098 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:16.060107 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:16.060112 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:44:16.060129 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:16.060172 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:44:16.060192 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:16.060309 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:16.060414 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 10:44:16.060472 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 10:44:16.061684 12161 log.cc:824] T 00000000000000000000000000000000 P 7b3249f1ad3b4dfb8051a01e9331aaaa: Log is configured to *not* fsync() on all Append() calls
I20260925 10:44:16.064301 12161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b3249f1ad3b4dfb8051a01e9331aaaa: No bootstrap required, opened a new log
I20260925 10:44:16.071509 12161 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b3249f1ad3b4dfb8051a01e9331aaaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b3249f1ad3b4dfb8051a01e9331aaaa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43055 } }
I20260925 10:44:16.071883 12161 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b3249f1ad3b4dfb8051a01e9331aaaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:44:16.071995 12161 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b3249f1ad3b4dfb8051a01e9331aaaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b3249f1ad3b4dfb8051a01e9331aaaa, State: Initialized, Role: FOLLOWER
I20260925 10:44:16.072600 12161 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b3249f1ad3b4dfb8051a01e9331aaaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b3249f1ad3b4dfb8051a01e9331aaaa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43055 } }
I20260925 10:44:16.072901 12161 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b3249f1ad3b4dfb8051a01e9331aaaa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:44:16.073077 12161 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b3249f1ad3b4dfb8051a01e9331aaaa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:44:16.073292 12161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b3249f1ad3b4dfb8051a01e9331aaaa [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:44:16.075541 12161 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b3249f1ad3b4dfb8051a01e9331aaaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b3249f1ad3b4dfb8051a01e9331aaaa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43055 } }
I20260925 10:44:16.076032 12161 leader_election.cc:302] T 00000000000000000000000000000000 P 7b3249f1ad3b4dfb8051a01e9331aaaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b3249f1ad3b4dfb8051a01e9331aaaa; no voters:
I20260925 10:44:16.076584 12161 leader_election.cc:288] T 00000000000000000000000000000000 P 7b3249f1ad3b4dfb8051a01e9331aaaa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:44:16.076723 12171 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b3249f1ad3b4dfb8051a01e9331aaaa [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:44:16.077375 12171 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b3249f1ad3b4dfb8051a01e9331aaaa [term 1 LEADER]: Becoming Leader. State: Replica: 7b3249f1ad3b4dfb8051a01e9331aaaa, State: Running, Role: LEADER
I20260925 10:44:16.077875 12171 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b3249f1ad3b4dfb8051a01e9331aaaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b3249f1ad3b4dfb8051a01e9331aaaa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43055 } }
I20260925 10:44:16.078645 12161 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b3249f1ad3b4dfb8051a01e9331aaaa [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:44:16.080506 12173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b3249f1ad3b4dfb8051a01e9331aaaa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b3249f1ad3b4dfb8051a01e9331aaaa. Latest consensus state: current_term: 1 leader_uuid: "7b3249f1ad3b4dfb8051a01e9331aaaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b3249f1ad3b4dfb8051a01e9331aaaa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43055 } } }
I20260925 10:44:16.081015 12173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b3249f1ad3b4dfb8051a01e9331aaaa [sys.catalog]: This master's current role is: LEADER
I20260925 10:44:16.080950 12172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b3249f1ad3b4dfb8051a01e9331aaaa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b3249f1ad3b4dfb8051a01e9331aaaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b3249f1ad3b4dfb8051a01e9331aaaa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43055 } } }
I20260925 10:44:16.081359 12172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b3249f1ad3b4dfb8051a01e9331aaaa [sys.catalog]: This master's current role is: LEADER
I20260925 10:44:16.082058 12178 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:44:16.087457 12178 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:44:16.088296 12188 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 10:44:16.096035 12178 catalog_manager.cc:1347] Generated new cluster ID: 0b6fa99075df4c649087c6799724139b
I20260925 10:44:16.096150 12178 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:44:16.108225 12178 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:44:16.108700 12178 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:44:16.115648 12178 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b3249f1ad3b4dfb8051a01e9331aaaa: Generated new TSK 0
I20260925 10:44:16.116421 12178 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:44:16.122228 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 12094
W20260925 10:44:16.123656 12156 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54211
W20260925 10:44:16.124312 23952 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 12094: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:44:16.124467 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 12094
2026-09-25T10:44:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1830 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790332740062128-23952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:44:16 dist-test-slave-dxdd krb5kdc[12192](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:44:16 dist-test-slave-dxdd krb5kdc[12192](info): set up 2 sockets
Sep 25 10:44:16 dist-test-slave-dxdd krb5kdc[12192](info): commencing operation
krb5kdc: starting...
W20260925 10:44:17.206839 23952 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Sep 25 10:44:17 dist-test-slave-dxdd krb5kdc[12192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:44:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:44:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790332740062128-23952-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790332740062128-23952-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:44:17.450192 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45235
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790332740062128-23952-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45235
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:44:17.764261 12205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:44:17.764518 12205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:44:17.764690 12205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:44:17.771495 12205 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:44:17.771621 12205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:44:17.771725 12205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:44:17.771842 12205 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:44:17.771926 12205 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:44:17.780989 12205 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790332740062128-23952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45235
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790332740062128-23952-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:44:17.782218 12205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:44:17.783519 12205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:44:17.795382 12210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:17.795307 12211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:17.795414 12213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:44:17.796060 12205 server_base.cc:1034] running on GCE node
I20260925 10:44:17.796838 12205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:44:17.798156 12205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:44:17.799372 12205 hybrid_clock.cc:648] HybridClock initialized: now 1790333057799336 us; error 51 us; skew 500 ppm
I20260925 10:44:17.799713 12205 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:44:17 dist-test-slave-dxdd krb5kdc[12192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:44:17.808171 12205 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:44:17.809698 12205 webserver.cc:460] Webserver started at http://127.0.0.1:34893/ using document root <none> and password file <none>
I20260925 10:44:17.810424 12205 fs_manager.cc:359] Metadata directory not provided
I20260925 10:44:17.810571 12205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:44:17.810869 12205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:44:17.813670 12205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790332740062128-23952-0/minicluster-data/master-0/data/instance:
uuid: "2d25049450de43fa9af215763b3ae7c2"
format_stamp: "Formatted at 2026-09-25 10:44:17 on dist-test-slave-dxdd"
server_key: "2e7e075c75274c677da395fec2e59cc8"
server_key_iv: "e53220195e519b9b6f02de140db95b05"
server_key_version: "encryptionkey@0"
I20260925 10:44:17.814448 12205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790332740062128-23952-0/minicluster-data/master-0/wal/instance:
uuid: "2d25049450de43fa9af215763b3ae7c2"
format_stamp: "Formatted at 2026-09-25 10:44:17 on dist-test-slave-dxdd"
server_key: "2e7e075c75274c677da395fec2e59cc8"
server_key_iv: "e53220195e519b9b6f02de140db95b05"
server_key_version: "encryptionkey@0"
I20260925 10:44:17.819208 12205 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 10:44:17.822430 12220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:44:17.823940 12205 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 10:44:17.824143 12205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790332740062128-23952-0/minicluster-data/master-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790332740062128-23952-0/minicluster-data/master-0/wal
uuid: "2d25049450de43fa9af215763b3ae7c2"
format_stamp: "Formatted at 2026-09-25 10:44:17 on dist-test-slave-dxdd"
server_key: "2e7e075c75274c677da395fec2e59cc8"
server_key_iv: "e53220195e519b9b6f02de140db95b05"
server_key_version: "encryptionkey@0"
I20260925 10:44:17.824342 12205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790332740062128-23952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790332740062128-23952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790332740062128-23952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:44:17.842394 12205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:44:17.845933 12205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:44:17.846195 12205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:44:17.863701 12205 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45235
I20260925 10:44:17.863739 12271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45235 every 8 connection(s)
I20260925 10:44:17.864964 12205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
I20260925 10:44:17.868901 12272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:44:17.872849 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 12205
I20260925 10:44:17.873036 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790332740062128-23952-0/minicluster-data/master-0/wal/instance
I20260925 10:44:17.873569 23952 external_mini_cluster.cc:1428] Setting key 04542d765f0d664d5789bfd4e8cfb6e2
I20260925 10:44:17.881588 12272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d25049450de43fa9af215763b3ae7c2: Bootstrap starting.
I20260925 10:44:17.885118 12272 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d25049450de43fa9af215763b3ae7c2: Neither blocks nor log segments found. Creating new log.
I20260925 10:44:17.886464 12272 log.cc:824] T 00000000000000000000000000000000 P 2d25049450de43fa9af215763b3ae7c2: Log is configured to *not* fsync() on all Append() calls
Sep 25 10:44:17 dist-test-slave-dxdd krb5kdc[12192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 10:44:17.889639 12272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d25049450de43fa9af215763b3ae7c2: No bootstrap required, opened a new log
I20260925 10:44:17.897260 12275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:17.875211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38591 (local address 127.0.0.1:45235)
0925 10:44:17.876000 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:17.876010 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:17.876047 (+ 37us) server_negotiation.cc:407] Connection header received
0925 10:44:17.877645 (+ 1598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:17.877664 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:17.878024 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:17.878362 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:17.880106 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:17.881335 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:17.883129 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:17.883744 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:17.883869 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:17.888962 (+ 5093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:17.888974 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:17.888993 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:44:17.889038 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:17.893012 (+ 3974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:17.893675 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:17.893684 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:17.893696 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:17.893902 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:17.894240 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:17.894246 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:17.894254 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:17.894921 (+ 667us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:17.895226 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:17.895555 (+ 329us) server_negotiation.cc:299] Negotiation successful
0925 10:44:17.895977 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":351,"threads_started":1}
I20260925 10:44:17.900203 12272 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d25049450de43fa9af215763b3ae7c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d25049450de43fa9af215763b3ae7c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45235 } }
I20260925 10:44:17.900581 12272 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d25049450de43fa9af215763b3ae7c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:44:17.900715 12272 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d25049450de43fa9af215763b3ae7c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d25049450de43fa9af215763b3ae7c2, State: Initialized, Role: FOLLOWER
I20260925 10:44:17.900914 23952 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:44:17.901320 12272 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d25049450de43fa9af215763b3ae7c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d25049450de43fa9af215763b3ae7c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45235 } }
I20260925 10:44:17.901561 12272 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d25049450de43fa9af215763b3ae7c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:44:17.901701 12272 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d25049450de43fa9af215763b3ae7c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:44:17.901865 12272 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d25049450de43fa9af215763b3ae7c2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:44:17.903947 12272 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d25049450de43fa9af215763b3ae7c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d25049450de43fa9af215763b3ae7c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45235 } }
I20260925 10:44:17.904441 12272 leader_election.cc:302] T 00000000000000000000000000000000 P 2d25049450de43fa9af215763b3ae7c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d25049450de43fa9af215763b3ae7c2; no voters:
I20260925 10:44:17.905010 12272 leader_election.cc:288] T 00000000000000000000000000000000 P 2d25049450de43fa9af215763b3ae7c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:44:17.905143 12281 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d25049450de43fa9af215763b3ae7c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:44:17.906013 12281 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d25049450de43fa9af215763b3ae7c2 [term 1 LEADER]: Becoming Leader. State: Replica: 2d25049450de43fa9af215763b3ae7c2, State: Running, Role: LEADER
I20260925 10:44:17.906663 12281 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d25049450de43fa9af215763b3ae7c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d25049450de43fa9af215763b3ae7c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45235 } }
I20260925 10:44:17.908212 12272 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d25049450de43fa9af215763b3ae7c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:44:17.909461 12282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d25049450de43fa9af215763b3ae7c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d25049450de43fa9af215763b3ae7c2. Latest consensus state: current_term: 1 leader_uuid: "2d25049450de43fa9af215763b3ae7c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d25049450de43fa9af215763b3ae7c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45235 } } }
I20260925 10:44:17.909492 12283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d25049450de43fa9af215763b3ae7c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d25049450de43fa9af215763b3ae7c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d25049450de43fa9af215763b3ae7c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45235 } } }
I20260925 10:44:17.909951 12283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d25049450de43fa9af215763b3ae7c2 [sys.catalog]: This master's current role is: LEADER
I20260925 10:44:17.909951 12282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d25049450de43fa9af215763b3ae7c2 [sys.catalog]: This master's current role is: LEADER
I20260925 10:44:17.928299 12275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:17.905502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39729 (local address 127.0.0.1:45235)
0925 10:44:17.905716 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:17.905724 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:17.907220 (+ 1496us) server_negotiation.cc:407] Connection header received
0925 10:44:17.907371 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:17.907378 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:17.907458 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:17.907558 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:17.909457 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:17.911125 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:17.913517 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:17.914517 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:17.914639 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:17.918687 (+ 4048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:17.918697 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:17.918705 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:44:17.918752 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:17.923335 (+ 4583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:17.924466 (+ 1131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:17.924473 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:17.924479 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:17.924639 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:17.925170 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:17.925177 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:17.925183 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:17.925748 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:17.927372 (+ 1624us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:17.927539 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 10:44:17.927821 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 10:44:17.930415 12287 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:44:17.936689 12287 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:44:17.945261 12287 catalog_manager.cc:1347] Generated new cluster ID: 98fc1241a57b4e7e938bdc8597257589
I20260925 10:44:17.945375 12287 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:44:17.961941 12287 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:44:17.963176 12287 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:44:17.969566 12287 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d25049450de43fa9af215763b3ae7c2: Generated new TSK 0
I20260925 10:44:17.970260 12287 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:44:17.975322 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 12205
I20260925 10:44:17.976773 12267 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:44:18.064173 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 12205
2026-09-25T10:44:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1943 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790332740062128-23952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:44:18 dist-test-slave-dxdd krb5kdc[12304](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:44:18 dist-test-slave-dxdd krb5kdc[12304](info): set up 2 sockets
Sep 25 10:44:18 dist-test-slave-dxdd krb5kdc[12304](info): commencing operation
krb5kdc: starting...
W20260925 10:44:19.162865 23952 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Sep 25 10:44:19 dist-test-slave-dxdd krb5kdc[12304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:44:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:44:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790332740062128-23952-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790332740062128-23952-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:44:19.392822 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34321
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790332740062128-23952-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34321
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:44:19.710866 12317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:44:19.711124 12317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:44:19.711314 12317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:44:19.717828 12317 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:44:19.717916 12317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:44:19.718044 12317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:44:19.718139 12317 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:44:19.718202 12317 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:44:19.727941 12317 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790332740062128-23952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34321
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790332740062128-23952-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:44:19.729025 12317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:44:19.730237 12317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:44:19.741639 12322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:19.741595 12323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:19.741683 12325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:44:19.742411 12317 server_base.cc:1034] running on GCE node
I20260925 10:44:19.743120 12317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:44:19.744415 12317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:44:19.745628 12317 hybrid_clock.cc:648] HybridClock initialized: now 1790333059745600 us; error 48 us; skew 500 ppm
I20260925 10:44:19.745947 12317 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:44:19 dist-test-slave-dxdd krb5kdc[12304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333059, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:44:19.748819 12317 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:44:19.750232 12317 webserver.cc:460] Webserver started at http://127.0.0.1:33819/ using document root <none> and password file <none>
I20260925 10:44:19.750903 12317 fs_manager.cc:359] Metadata directory not provided
I20260925 10:44:19.751049 12317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:44:19.751341 12317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:44:19.754076 12317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790332740062128-23952-0/minicluster-data/master-0/data/instance:
uuid: "a7f527763d944ccb9ee02b762f0c355c"
format_stamp: "Formatted at 2026-09-25 10:44:19 on dist-test-slave-dxdd"
server_key: "1b9702e156e943e23e3eb526fff1138c"
server_key_iv: "f17be4c9b6160194feb41cfb053e7d96"
server_key_version: "encryptionkey@0"
I20260925 10:44:19.754868 12317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790332740062128-23952-0/minicluster-data/master-0/wal/instance:
uuid: "a7f527763d944ccb9ee02b762f0c355c"
format_stamp: "Formatted at 2026-09-25 10:44:19 on dist-test-slave-dxdd"
server_key: "1b9702e156e943e23e3eb526fff1138c"
server_key_iv: "f17be4c9b6160194feb41cfb053e7d96"
server_key_version: "encryptionkey@0"
I20260925 10:44:19.759690 12317 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260925 10:44:19.762989 12332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:44:19.764262 12317 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 10:44:19.764528 12317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790332740062128-23952-0/minicluster-data/master-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790332740062128-23952-0/minicluster-data/master-0/wal
uuid: "a7f527763d944ccb9ee02b762f0c355c"
format_stamp: "Formatted at 2026-09-25 10:44:19 on dist-test-slave-dxdd"
server_key: "1b9702e156e943e23e3eb526fff1138c"
server_key_iv: "f17be4c9b6160194feb41cfb053e7d96"
server_key_version: "encryptionkey@0"
I20260925 10:44:19.764721 12317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790332740062128-23952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790332740062128-23952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790332740062128-23952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:44:19.785442 12317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:44:19.789173 12317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:44:19.789466 12317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:44:19.807034 12317 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34321
I20260925 10:44:19.807076 12383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34321 every 8 connection(s)
I20260925 10:44:19.808296 12317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
I20260925 10:44:19.811735 12384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:44:19.816303 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 12317
I20260925 10:44:19.816483 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790332740062128-23952-0/minicluster-data/master-0/wal/instance
I20260925 10:44:19.817039 23952 external_mini_cluster.cc:1428] Setting key 31bd28cb7cc369c814149f0cd5db39a6
I20260925 10:44:19.822678 12384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7f527763d944ccb9ee02b762f0c355c: Bootstrap starting.
I20260925 10:44:19.825590 12384 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7f527763d944ccb9ee02b762f0c355c: Neither blocks nor log segments found. Creating new log.
I20260925 10:44:19.826717 12384 log.cc:824] T 00000000000000000000000000000000 P a7f527763d944ccb9ee02b762f0c355c: Log is configured to *not* fsync() on all Append() calls
I20260925 10:44:19.829756 12384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7f527763d944ccb9ee02b762f0c355c: No bootstrap required, opened a new log
I20260925 10:44:19.829872 12387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:19.818790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59085 (local address 127.0.0.1:34321)
0925 10:44:19.819616 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:19.819626 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:19.819666 (+ 40us) server_negotiation.cc:407] Connection header received
0925 10:44:19.821445 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:19.821465 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:19.821806 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:19.822149 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:19.822667 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:19.824011 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:19.825304 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:19.826038 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:19.826230 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:19.827408 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:19.827419 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:19.827432 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:44:19.827985 (+ 553us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:19.828082 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:44:19.828111 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:19.828224 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:19.828436 (+ 212us) server_negotiation.cc:299] Negotiation successful
0925 10:44:19.828667 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":427,"threads_started":1}
I20260925 10:44:19.833312 23952 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:44:19.837728 12384 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7f527763d944ccb9ee02b762f0c355c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f527763d944ccb9ee02b762f0c355c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34321 } }
I20260925 10:44:19.838172 12384 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7f527763d944ccb9ee02b762f0c355c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:44:19.838299 12384 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7f527763d944ccb9ee02b762f0c355c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7f527763d944ccb9ee02b762f0c355c, State: Initialized, Role: FOLLOWER
I20260925 10:44:19.839030 12384 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7f527763d944ccb9ee02b762f0c355c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f527763d944ccb9ee02b762f0c355c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34321 } }
I20260925 10:44:19.839352 12384 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7f527763d944ccb9ee02b762f0c355c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:44:19.839515 12384 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7f527763d944ccb9ee02b762f0c355c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:44:19.839694 12384 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7f527763d944ccb9ee02b762f0c355c [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:44:19.842644 12384 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7f527763d944ccb9ee02b762f0c355c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f527763d944ccb9ee02b762f0c355c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34321 } }
I20260925 10:44:19.843202 12384 leader_election.cc:302] T 00000000000000000000000000000000 P a7f527763d944ccb9ee02b762f0c355c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7f527763d944ccb9ee02b762f0c355c; no voters:
I20260925 10:44:19.843950 12384 leader_election.cc:288] T 00000000000000000000000000000000 P a7f527763d944ccb9ee02b762f0c355c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:44:19.844053 12394 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7f527763d944ccb9ee02b762f0c355c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:44:19.844507 12387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:19.837590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50139 (local address 127.0.0.1:34321)
0925 10:44:19.837801 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:19.837809 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:19.838296 (+ 487us) server_negotiation.cc:407] Connection header received
0925 10:44:19.838471 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:19.838477 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:19.838546 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:19.838624 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:19.839217 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:19.840130 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:19.841946 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:19.842440 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:19.842525 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:19.843601 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:19.843605 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:19.843608 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:44:19.843623 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:19.843656 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:44:19.843669 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:19.843779 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:19.843945 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 10:44:19.844022 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 10:44:19.846486 12384 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7f527763d944ccb9ee02b762f0c355c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:44:19.846573 12394 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7f527763d944ccb9ee02b762f0c355c [term 1 LEADER]: Becoming Leader. State: Replica: a7f527763d944ccb9ee02b762f0c355c, State: Running, Role: LEADER
I20260925 10:44:19.847162 12394 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7f527763d944ccb9ee02b762f0c355c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f527763d944ccb9ee02b762f0c355c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34321 } }
I20260925 10:44:19.850083 12395 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7f527763d944ccb9ee02b762f0c355c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7f527763d944ccb9ee02b762f0c355c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f527763d944ccb9ee02b762f0c355c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34321 } } }
I20260925 10:44:19.850199 12396 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7f527763d944ccb9ee02b762f0c355c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7f527763d944ccb9ee02b762f0c355c. Latest consensus state: current_term: 1 leader_uuid: "a7f527763d944ccb9ee02b762f0c355c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f527763d944ccb9ee02b762f0c355c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34321 } } }
I20260925 10:44:19.850554 12395 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7f527763d944ccb9ee02b762f0c355c [sys.catalog]: This master's current role is: LEADER
I20260925 10:44:19.850630 12396 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7f527763d944ccb9ee02b762f0c355c [sys.catalog]: This master's current role is: LEADER
I20260925 10:44:19.853440 12399 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:44:19.857664 12399 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:44:19.865584 12399 catalog_manager.cc:1347] Generated new cluster ID: 2dd942b738db40ee95010d42fd510355
I20260925 10:44:19.865675 12399 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:44:19.880954 12399 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:44:19.882756 12399 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:44:19.898291 12399 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7f527763d944ccb9ee02b762f0c355c: Generated new TSK 0
I20260925 10:44:19.899338 12399 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:44:19.910790 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 12317
W20260925 10:44:19.911762 12379 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59085
W20260925 10:44:19.912437 23952 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 12317: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:44:19.912595 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 12317
2026-09-25T10:44:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1845 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790332740062128-23952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:44:20 dist-test-slave-dxdd krb5kdc[12415](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:44:20 dist-test-slave-dxdd krb5kdc[12415](info): set up 2 sockets
Sep 25 10:44:20 dist-test-slave-dxdd krb5kdc[12415](info): commencing operation
krb5kdc: starting...
W20260925 10:44:20.990783 23952 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Sep 25 10:44:21 dist-test-slave-dxdd krb5kdc[12415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:44:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:44:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790332740062128-23952-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790332740062128-23952-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:44:21.241431 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41563
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790332740062128-23952-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41563
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:44:21.558286 12428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:44:21.558640 12428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:44:21.558791 12428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:44:21.565971 12428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:44:21.566115 12428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:44:21.566226 12428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:44:21.566354 12428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:44:21.566432 12428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:44:21.575763 12428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790332740062128-23952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41563
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790332740062128-23952-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:44:21.577009 12428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:44:21.578255 12428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:44:21.589608 12434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:21.589828 12436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:21.589715 12433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:44:21.590598 12428 server_base.cc:1034] running on GCE node
I20260925 10:44:21.591251 12428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:44:21.592588 12428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:44:21.593818 12428 hybrid_clock.cc:648] HybridClock initialized: now 1790333061593766 us; error 44 us; skew 500 ppm
I20260925 10:44:21.594120 12428 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:44:21 dist-test-slave-dxdd krb5kdc[12415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333061, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:44:21.597254 12428 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:44:21.598692 12428 webserver.cc:460] Webserver started at http://127.0.0.1:45257/ using document root <none> and password file <none>
I20260925 10:44:21.599352 12428 fs_manager.cc:359] Metadata directory not provided
I20260925 10:44:21.599483 12428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:44:21.599790 12428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:44:21.602778 12428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790332740062128-23952-0/minicluster-data/master-0/data/instance:
uuid: "8cfc7c09a25f40ba8e59c948f313eb08"
format_stamp: "Formatted at 2026-09-25 10:44:21 on dist-test-slave-dxdd"
server_key: "ea5849cc5047dc0245681e5be92084e2"
server_key_iv: "426337f2e29a1558a78dd8f24dcfccc3"
server_key_version: "encryptionkey@0"
I20260925 10:44:21.603554 12428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790332740062128-23952-0/minicluster-data/master-0/wal/instance:
uuid: "8cfc7c09a25f40ba8e59c948f313eb08"
format_stamp: "Formatted at 2026-09-25 10:44:21 on dist-test-slave-dxdd"
server_key: "ea5849cc5047dc0245681e5be92084e2"
server_key_iv: "426337f2e29a1558a78dd8f24dcfccc3"
server_key_version: "encryptionkey@0"
I20260925 10:44:21.608697 12428 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 10:44:21.612133 12443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:44:21.613523 12428 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 10:44:21.613708 12428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790332740062128-23952-0/minicluster-data/master-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790332740062128-23952-0/minicluster-data/master-0/wal
uuid: "8cfc7c09a25f40ba8e59c948f313eb08"
format_stamp: "Formatted at 2026-09-25 10:44:21 on dist-test-slave-dxdd"
server_key: "ea5849cc5047dc0245681e5be92084e2"
server_key_iv: "426337f2e29a1558a78dd8f24dcfccc3"
server_key_version: "encryptionkey@0"
I20260925 10:44:21.613935 12428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790332740062128-23952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790332740062128-23952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790332740062128-23952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:44:21.619407 12428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:44:21.623777 12428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:44:21.624081 12428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:44:21.642102 12428 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41563
I20260925 10:44:21.642149 12494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41563 every 8 connection(s)
I20260925 10:44:21.643483 12428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
I20260925 10:44:21.647104 12495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:44:21.653499 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 12428
I20260925 10:44:21.653677 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790332740062128-23952-0/minicluster-data/master-0/wal/instance
I20260925 10:44:21.654089 23952 external_mini_cluster.cc:1428] Setting key c07263e67a6df6286f423471c30aaec8
I20260925 10:44:21.657243 12495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cfc7c09a25f40ba8e59c948f313eb08: Bootstrap starting.
I20260925 10:44:21.660718 12495 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8cfc7c09a25f40ba8e59c948f313eb08: Neither blocks nor log segments found. Creating new log.
I20260925 10:44:21.661643 12498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:21.655873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51703 (local address 127.0.0.1:41563)
0925 10:44:21.656691 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:21.656715 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:21.656753 (+ 38us) server_negotiation.cc:407] Connection header received
0925 10:44:21.658421 (+ 1668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:21.658442 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:21.658761 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:21.659092 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:21.660132 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:21.660138 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:21.660150 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:44:21.660163 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:21.660220 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:44:21.660238 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:21.660371 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:21.660548 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 10:44:21.660738 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":352,"threads_started":1}
I20260925 10:44:21.661865 12495 log.cc:824] T 00000000000000000000000000000000 P 8cfc7c09a25f40ba8e59c948f313eb08: Log is configured to *not* fsync() on all Append() calls
I20260925 10:44:21.664994 12495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cfc7c09a25f40ba8e59c948f313eb08: No bootstrap required, opened a new log
I20260925 10:44:21.665531 23952 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:44:21.672979 12498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:21.670583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60603 (local address 127.0.0.1:41563)
0925 10:44:21.670801 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:21.670807 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:21.671159 (+ 352us) server_negotiation.cc:407] Connection header received
0925 10:44:21.671309 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:21.671312 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:21.671353 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:21.671433 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:21.672328 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:21.672332 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:21.672335 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:44:21.672351 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:21.672388 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:44:21.672403 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:21.672502 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:21.672619 (+ 117us) server_negotiation.cc:299] Negotiation successful
0925 10:44:21.672667 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 10:44:21.675339 12495 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8cfc7c09a25f40ba8e59c948f313eb08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cfc7c09a25f40ba8e59c948f313eb08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41563 } }
I20260925 10:44:21.675680 12495 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8cfc7c09a25f40ba8e59c948f313eb08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:44:21.675799 12495 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8cfc7c09a25f40ba8e59c948f313eb08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cfc7c09a25f40ba8e59c948f313eb08, State: Initialized, Role: FOLLOWER
I20260925 10:44:21.676334 12495 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8cfc7c09a25f40ba8e59c948f313eb08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cfc7c09a25f40ba8e59c948f313eb08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41563 } }
I20260925 10:44:21.676579 12495 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8cfc7c09a25f40ba8e59c948f313eb08 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:44:21.676735 12495 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8cfc7c09a25f40ba8e59c948f313eb08 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:44:21.676909 12495 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8cfc7c09a25f40ba8e59c948f313eb08 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:44:21.679124 12495 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8cfc7c09a25f40ba8e59c948f313eb08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cfc7c09a25f40ba8e59c948f313eb08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41563 } }
I20260925 10:44:21.679610 12495 leader_election.cc:302] T 00000000000000000000000000000000 P 8cfc7c09a25f40ba8e59c948f313eb08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8cfc7c09a25f40ba8e59c948f313eb08; no voters:
I20260925 10:44:21.680126 12495 leader_election.cc:288] T 00000000000000000000000000000000 P 8cfc7c09a25f40ba8e59c948f313eb08 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:44:21.680265 12505 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8cfc7c09a25f40ba8e59c948f313eb08 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:44:21.681077 12505 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8cfc7c09a25f40ba8e59c948f313eb08 [term 1 LEADER]: Becoming Leader. State: Replica: 8cfc7c09a25f40ba8e59c948f313eb08, State: Running, Role: LEADER
I20260925 10:44:21.681574 12505 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8cfc7c09a25f40ba8e59c948f313eb08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cfc7c09a25f40ba8e59c948f313eb08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41563 } }
I20260925 10:44:21.682230 12495 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8cfc7c09a25f40ba8e59c948f313eb08 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:44:21.684249 12506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cfc7c09a25f40ba8e59c948f313eb08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8cfc7c09a25f40ba8e59c948f313eb08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cfc7c09a25f40ba8e59c948f313eb08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41563 } } }
I20260925 10:44:21.684643 12506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cfc7c09a25f40ba8e59c948f313eb08 [sys.catalog]: This master's current role is: LEADER
I20260925 10:44:21.686569 12507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cfc7c09a25f40ba8e59c948f313eb08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8cfc7c09a25f40ba8e59c948f313eb08. Latest consensus state: current_term: 1 leader_uuid: "8cfc7c09a25f40ba8e59c948f313eb08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cfc7c09a25f40ba8e59c948f313eb08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41563 } } }
I20260925 10:44:21.687472 12507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cfc7c09a25f40ba8e59c948f313eb08 [sys.catalog]: This master's current role is: LEADER
I20260925 10:44:21.688264 12510 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:44:21.693897 12510 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:44:21.702464 12510 catalog_manager.cc:1347] Generated new cluster ID: cf0fd0f5878a4526ada6eea178b5a928
I20260925 10:44:21.702569 12510 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:44:21.730225 12510 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:44:21.730649 12510 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:44:21.736842 12510 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8cfc7c09a25f40ba8e59c948f313eb08: Generated new TSK 0
I20260925 10:44:21.737579 12510 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:44:21.747599 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 12428
W20260925 10:44:21.748657 12490 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51703
W20260925 10:44:21.749360 23952 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 12428: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:44:21.749508 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 12428
2026-09-25T10:44:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1836 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790332740062128-23952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:44:21 dist-test-slave-dxdd krb5kdc[12526](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:44:21 dist-test-slave-dxdd krb5kdc[12526](info): set up 2 sockets
Sep 25 10:44:21 dist-test-slave-dxdd krb5kdc[12526](info): commencing operation
krb5kdc: starting...
W20260925 10:44:22.838845 23952 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Sep 25 10:44:22 dist-test-slave-dxdd krb5kdc[12526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:44:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:44:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790332740062128-23952-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790332740062128-23952-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:44:23.065799 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35519
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790332740062128-23952-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35519
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:44:23.385162 12539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:44:23.385545 12539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:44:23.385673 12539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:44:23.392930 12539 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:44:23.393026 12539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:44:23.393071 12539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:44:23.393128 12539 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:44:23.393164 12539 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:44:23.402697 12539 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790332740062128-23952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790332740062128-23952-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:44:23.403831 12539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:44:23.405017 12539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:44:23.416371 12544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:23.416483 12547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:23.416532 12545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:44:23.417380 12539 server_base.cc:1034] running on GCE node
I20260925 10:44:23.418156 12539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:44:23.419538 12539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:44:23.420768 12539 hybrid_clock.cc:648] HybridClock initialized: now 1790333063420732 us; error 52 us; skew 500 ppm
I20260925 10:44:23.421144 12539 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:44:23 dist-test-slave-dxdd krb5kdc[12526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333063, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:44:23.429546 12539 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:44:23.431082 12539 webserver.cc:460] Webserver started at http://127.0.0.1:37081/ using document root <none> and password file <none>
I20260925 10:44:23.431753 12539 fs_manager.cc:359] Metadata directory not provided
I20260925 10:44:23.431898 12539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:44:23.432191 12539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:44:23.435074 12539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790332740062128-23952-0/minicluster-data/master-0/data/instance:
uuid: "ac76763865c5415380b5b98c46e25ecf"
format_stamp: "Formatted at 2026-09-25 10:44:23 on dist-test-slave-dxdd"
server_key: "301f72281bcca1548ff414bd8528c51a"
server_key_iv: "0cec9e80d55f246ce787522d69ab4d96"
server_key_version: "encryptionkey@0"
I20260925 10:44:23.435853 12539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790332740062128-23952-0/minicluster-data/master-0/wal/instance:
uuid: "ac76763865c5415380b5b98c46e25ecf"
format_stamp: "Formatted at 2026-09-25 10:44:23 on dist-test-slave-dxdd"
server_key: "301f72281bcca1548ff414bd8528c51a"
server_key_iv: "0cec9e80d55f246ce787522d69ab4d96"
server_key_version: "encryptionkey@0"
I20260925 10:44:23.440841 12539 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 10:44:23.444249 12554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:44:23.445513 12539 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 10:44:23.445695 12539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790332740062128-23952-0/minicluster-data/master-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790332740062128-23952-0/minicluster-data/master-0/wal
uuid: "ac76763865c5415380b5b98c46e25ecf"
format_stamp: "Formatted at 2026-09-25 10:44:23 on dist-test-slave-dxdd"
server_key: "301f72281bcca1548ff414bd8528c51a"
server_key_iv: "0cec9e80d55f246ce787522d69ab4d96"
server_key_version: "encryptionkey@0"
I20260925 10:44:23.445883 12539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790332740062128-23952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790332740062128-23952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790332740062128-23952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:44:23.468127 12539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:44:23.472047 12539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:44:23.472330 12539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:44:23.490360 12539 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35519
I20260925 10:44:23.490379 12605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35519 every 8 connection(s)
I20260925 10:44:23.491667 12539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
I20260925 10:44:23.495093 12606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:44:23.500535 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 12539
I20260925 10:44:23.500697 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790332740062128-23952-0/minicluster-data/master-0/wal/instance
I20260925 10:44:23.501163 23952 external_mini_cluster.cc:1428] Setting key 1a35580231e68b7ea5de3e97af02ef30
I20260925 10:44:23.506235 12606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac76763865c5415380b5b98c46e25ecf: Bootstrap starting.
I20260925 10:44:23.509793 12606 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac76763865c5415380b5b98c46e25ecf: Neither blocks nor log segments found. Creating new log.
I20260925 10:44:23.510972 12606 log.cc:824] T 00000000000000000000000000000000 P ac76763865c5415380b5b98c46e25ecf: Log is configured to *not* fsync() on all Append() calls
I20260925 10:44:23.514071 12606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac76763865c5415380b5b98c46e25ecf: No bootstrap required, opened a new log
Sep 25 10:44:23 dist-test-slave-dxdd krb5kdc[12526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 10:44:23.523044 12606 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac76763865c5415380b5b98c46e25ecf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac76763865c5415380b5b98c46e25ecf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35519 } }
I20260925 10:44:23.523470 12606 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac76763865c5415380b5b98c46e25ecf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:44:23.523598 12606 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac76763865c5415380b5b98c46e25ecf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac76763865c5415380b5b98c46e25ecf, State: Initialized, Role: FOLLOWER
I20260925 10:44:23.524181 12606 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac76763865c5415380b5b98c46e25ecf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac76763865c5415380b5b98c46e25ecf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35519 } }
I20260925 10:44:23.524430 12606 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac76763865c5415380b5b98c46e25ecf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:44:23.524595 12606 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac76763865c5415380b5b98c46e25ecf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:44:23.524763 12606 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac76763865c5415380b5b98c46e25ecf [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:44:23.524993 12609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:23.503047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.183:48191 (local address 127.0.0.1:35519)
0925 10:44:23.503864 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:23.503879 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:23.503929 (+ 50us) server_negotiation.cc:407] Connection header received
0925 10:44:23.506255 (+ 2326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:23.506285 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:23.506787 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:23.507195 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:23.508444 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:23.510452 (+ 2008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:23.511991 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:23.512917 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:23.513105 (+ 188us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:23.517521 (+ 4416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:23.517531 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:23.517543 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:44:23.517579 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:23.520584 (+ 3005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:23.521479 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:23.521485 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:23.521490 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:23.521643 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:23.522131 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:23.522134 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:23.522136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:23.522653 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:23.522863 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:23.523409 (+ 546us) server_negotiation.cc:299] Negotiation successful
0925 10:44:23.523787 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":354,"threads_started":1}
I20260925 10:44:23.527173 12606 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac76763865c5415380b5b98c46e25ecf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac76763865c5415380b5b98c46e25ecf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35519 } }
I20260925 10:44:23.527733 12606 leader_election.cc:302] T 00000000000000000000000000000000 P ac76763865c5415380b5b98c46e25ecf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac76763865c5415380b5b98c46e25ecf; no voters:
I20260925 10:44:23.528344 12606 leader_election.cc:288] T 00000000000000000000000000000000 P ac76763865c5415380b5b98c46e25ecf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:44:23.528420 12611 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac76763865c5415380b5b98c46e25ecf [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:44:23.528784 23952 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:44:23.530175 12606 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac76763865c5415380b5b98c46e25ecf [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:44:23.533800 12611 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac76763865c5415380b5b98c46e25ecf [term 1 LEADER]: Becoming Leader. State: Replica: ac76763865c5415380b5b98c46e25ecf, State: Running, Role: LEADER
I20260925 10:44:23.534432 12611 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac76763865c5415380b5b98c46e25ecf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac76763865c5415380b5b98c46e25ecf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35519 } }
I20260925 10:44:23.542582 12613 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac76763865c5415380b5b98c46e25ecf [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac76763865c5415380b5b98c46e25ecf. Latest consensus state: current_term: 1 leader_uuid: "ac76763865c5415380b5b98c46e25ecf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac76763865c5415380b5b98c46e25ecf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35519 } } }
I20260925 10:44:23.542932 12613 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac76763865c5415380b5b98c46e25ecf [sys.catalog]: This master's current role is: LEADER
I20260925 10:44:23.543488 12612 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac76763865c5415380b5b98c46e25ecf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac76763865c5415380b5b98c46e25ecf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac76763865c5415380b5b98c46e25ecf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35519 } } }
I20260925 10:44:23.543871 12612 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac76763865c5415380b5b98c46e25ecf [sys.catalog]: This master's current role is: LEADER
I20260925 10:44:23.551807 12631 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:44:23.556586 12631 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:44:23.559007 12609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:23.535779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.183:53689 (local address 127.0.0.1:35519)
0925 10:44:23.535968 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:23.535977 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:23.536911 (+ 934us) server_negotiation.cc:407] Connection header received
0925 10:44:23.537122 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:23.537129 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:23.537224 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:23.537348 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:23.539426 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:23.541007 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:23.543271 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:23.544249 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:23.544413 (+ 164us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:23.548488 (+ 4075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:23.548497 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:23.548505 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:44:23.548554 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:23.553429 (+ 4875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:23.554708 (+ 1279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:23.554717 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:23.554723 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:23.554915 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:23.555600 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:23.555608 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:23.555614 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:23.556116 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:23.556462 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:23.557988 (+ 1526us) server_negotiation.cc:299] Negotiation successful
0925 10:44:23.558243 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 10:44:23.565579 12631 catalog_manager.cc:1347] Generated new cluster ID: 58975060cf0b43feb6cd7c22d1723d7b
I20260925 10:44:23.565677 12631 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:44:23.579881 12631 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:44:23.581028 12631 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:44:23.588142 12631 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac76763865c5415380b5b98c46e25ecf: Generated new TSK 0
I20260925 10:44:23.588857 12631 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:44:23.594728 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 12539
I20260925 10:44:23.596122 12601 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:44:23.682734 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 12539
W20260925 10:44:23.699425 12524 connection.cc:535] client connection to 127.0.0.1:35519 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35519): Connection reset by peer (error 104)
2026-09-25T10:44:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1933 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790332740062128-23952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:44:23 dist-test-slave-dxdd krb5kdc[12638](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:44:23 dist-test-slave-dxdd krb5kdc[12638](info): set up 2 sockets
Sep 25 10:44:23 dist-test-slave-dxdd krb5kdc[12638](info): commencing operation
krb5kdc: starting...
W20260925 10:44:24.762898 23952 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Sep 25 10:44:24 dist-test-slave-dxdd krb5kdc[12638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:44:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:44:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790332740062128-23952-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790332740062128-23952-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:44:25.005544 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34933
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790332740062128-23952-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34933
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:44:25.320523 12651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:44:25.320796 12651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:44:25.320968 12651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:44:25.328006 12651 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:44:25.328130 12651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:44:25.328215 12651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:44:25.328325 12651 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:44:25.328421 12651 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:44:25.337901 12651 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790332740062128-23952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34933
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790332740062128-23952-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:44:25.339148 12651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:44:25.340432 12651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:44:25.351845 12656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:25.351908 12659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:25.351876 12657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:44:25.352653 12651 server_base.cc:1034] running on GCE node
I20260925 10:44:25.353403 12651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:44:25.354733 12651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:44:25.355958 12651 hybrid_clock.cc:648] HybridClock initialized: now 1790333065355924 us; error 40 us; skew 500 ppm
I20260925 10:44:25.356341 12651 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:44:25 dist-test-slave-dxdd krb5kdc[12638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333065, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:44:25.359458 12651 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:44:25.360807 12651 webserver.cc:460] Webserver started at http://127.0.0.1:35389/ using document root <none> and password file <none>
I20260925 10:44:25.361449 12651 fs_manager.cc:359] Metadata directory not provided
I20260925 10:44:25.361596 12651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:44:25.361886 12651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:44:25.365059 12651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790332740062128-23952-0/minicluster-data/master-0/data/instance:
uuid: "60ece35e0ec54a3aa90f8e5fb39b03e4"
format_stamp: "Formatted at 2026-09-25 10:44:25 on dist-test-slave-dxdd"
server_key: "d61773c965925ee7ad869838968ae477"
server_key_iv: "657413bd1a13d46cd5e1c371b7fc636d"
server_key_version: "encryptionkey@0"
I20260925 10:44:25.365775 12651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790332740062128-23952-0/minicluster-data/master-0/wal/instance:
uuid: "60ece35e0ec54a3aa90f8e5fb39b03e4"
format_stamp: "Formatted at 2026-09-25 10:44:25 on dist-test-slave-dxdd"
server_key: "d61773c965925ee7ad869838968ae477"
server_key_iv: "657413bd1a13d46cd5e1c371b7fc636d"
server_key_version: "encryptionkey@0"
I20260925 10:44:25.370698 12651 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 10:44:25.373844 12666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:44:25.375046 12651 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 10:44:25.375224 12651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790332740062128-23952-0/minicluster-data/master-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790332740062128-23952-0/minicluster-data/master-0/wal
uuid: "60ece35e0ec54a3aa90f8e5fb39b03e4"
format_stamp: "Formatted at 2026-09-25 10:44:25 on dist-test-slave-dxdd"
server_key: "d61773c965925ee7ad869838968ae477"
server_key_iv: "657413bd1a13d46cd5e1c371b7fc636d"
server_key_version: "encryptionkey@0"
I20260925 10:44:25.375401 12651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790332740062128-23952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790332740062128-23952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790332740062128-23952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:44:25.396898 12651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:44:25.401245 12651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:44:25.401542 12651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:44:25.419466 12651 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34933
I20260925 10:44:25.419500 12717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34933 every 8 connection(s)
I20260925 10:44:25.420764 12651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
I20260925 10:44:25.424235 12718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:44:25.427578 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 12651
I20260925 10:44:25.427757 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790332740062128-23952-0/minicluster-data/master-0/wal/instance
I20260925 10:44:25.428192 23952 external_mini_cluster.cc:1428] Setting key fc3d59e34fb874cd87acb212bca0ce5d
I20260925 10:44:25.436860 12718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60ece35e0ec54a3aa90f8e5fb39b03e4: Bootstrap starting.
I20260925 10:44:25.440518 12718 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60ece35e0ec54a3aa90f8e5fb39b03e4: Neither blocks nor log segments found. Creating new log.
I20260925 10:44:25.441833 12718 log.cc:824] T 00000000000000000000000000000000 P 60ece35e0ec54a3aa90f8e5fb39b03e4: Log is configured to *not* fsync() on all Append() calls
I20260925 10:44:25.442456 12721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:25.429889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.183:38361 (local address 127.0.0.1:34933)
0925 10:44:25.430651 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:25.430665 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:25.430712 (+ 47us) server_negotiation.cc:407] Connection header received
0925 10:44:25.432507 (+ 1795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:25.432533 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:25.433020 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:25.433435 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:25.433917 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:25.435949 (+ 2032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:25.437323 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:25.438263 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:25.438484 (+ 221us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:25.439622 (+ 1138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:25.439634 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:25.439653 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:44:25.440334 (+ 681us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:25.440427 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:44:25.440460 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:25.440615 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:25.440946 (+ 331us) server_negotiation.cc:299] Negotiation successful
0925 10:44:25.441122 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":349,"threads_started":1}
I20260925 10:44:25.444680 12718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60ece35e0ec54a3aa90f8e5fb39b03e4: No bootstrap required, opened a new log
I20260925 10:44:25.446166 23952 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:44:25.454648 12718 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60ece35e0ec54a3aa90f8e5fb39b03e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ece35e0ec54a3aa90f8e5fb39b03e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34933 } }
I20260925 10:44:25.454993 12718 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60ece35e0ec54a3aa90f8e5fb39b03e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:44:25.455089 12718 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60ece35e0ec54a3aa90f8e5fb39b03e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ece35e0ec54a3aa90f8e5fb39b03e4, State: Initialized, Role: FOLLOWER
I20260925 10:44:25.455758 12718 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60ece35e0ec54a3aa90f8e5fb39b03e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ece35e0ec54a3aa90f8e5fb39b03e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34933 } }
I20260925 10:44:25.456049 12718 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60ece35e0ec54a3aa90f8e5fb39b03e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:44:25.456218 12718 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60ece35e0ec54a3aa90f8e5fb39b03e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:44:25.456393 12718 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60ece35e0ec54a3aa90f8e5fb39b03e4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:44:25.457962 12721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:25.451330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.183:35297 (local address 127.0.0.1:34933)
0925 10:44:25.451496 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:25.451501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:25.451884 (+ 383us) server_negotiation.cc:407] Connection header received
0925 10:44:25.452014 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:25.452019 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:25.452092 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:25.452198 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:25.452641 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:25.453802 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:25.455223 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:25.455855 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:25.455998 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:25.457188 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:25.457196 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:25.457206 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:44:25.457229 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:25.457286 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:44:25.457312 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:25.457420 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:25.457585 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 10:44:25.457638 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 10:44:25.459160 12718 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60ece35e0ec54a3aa90f8e5fb39b03e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ece35e0ec54a3aa90f8e5fb39b03e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34933 } }
I20260925 10:44:25.459640 12718 leader_election.cc:302] T 00000000000000000000000000000000 P 60ece35e0ec54a3aa90f8e5fb39b03e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60ece35e0ec54a3aa90f8e5fb39b03e4; no voters:
I20260925 10:44:25.460173 12718 leader_election.cc:288] T 00000000000000000000000000000000 P 60ece35e0ec54a3aa90f8e5fb39b03e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:44:25.460306 12728 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60ece35e0ec54a3aa90f8e5fb39b03e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:44:25.460987 12728 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60ece35e0ec54a3aa90f8e5fb39b03e4 [term 1 LEADER]: Becoming Leader. State: Replica: 60ece35e0ec54a3aa90f8e5fb39b03e4, State: Running, Role: LEADER
I20260925 10:44:25.461515 12728 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60ece35e0ec54a3aa90f8e5fb39b03e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ece35e0ec54a3aa90f8e5fb39b03e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34933 } }
I20260925 10:44:25.462109 12718 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60ece35e0ec54a3aa90f8e5fb39b03e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:44:25.464275 12730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60ece35e0ec54a3aa90f8e5fb39b03e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60ece35e0ec54a3aa90f8e5fb39b03e4. Latest consensus state: current_term: 1 leader_uuid: "60ece35e0ec54a3aa90f8e5fb39b03e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ece35e0ec54a3aa90f8e5fb39b03e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34933 } } }
I20260925 10:44:25.465183 12730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60ece35e0ec54a3aa90f8e5fb39b03e4 [sys.catalog]: This master's current role is: LEADER
I20260925 10:44:25.465150 12729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60ece35e0ec54a3aa90f8e5fb39b03e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60ece35e0ec54a3aa90f8e5fb39b03e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ece35e0ec54a3aa90f8e5fb39b03e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34933 } } }
I20260925 10:44:25.465435 12729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60ece35e0ec54a3aa90f8e5fb39b03e4 [sys.catalog]: This master's current role is: LEADER
W20260925 10:44:25.473363 12744 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 60ece35e0ec54a3aa90f8e5fb39b03e4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:44:25.473469 12744 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:44:25.473628 12736 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:44:25.475354 12736 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:44:25.482697 12736 catalog_manager.cc:1347] Generated new cluster ID: f22919466c404c9dbdde0e1b7a55ae41
I20260925 10:44:25.482806 12736 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:44:25.499054 12736 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:44:25.500792 12736 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:44:25.510969 12736 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60ece35e0ec54a3aa90f8e5fb39b03e4: Generated new TSK 0
I20260925 10:44:25.512048 12736 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:44:25.527633 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 12651
W20260925 10:44:25.528801 12713 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.183:38361
W20260925 10:44:25.529526 23952 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 12651: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:44:25.529714 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 12651
W20260925 10:44:25.546348 12636 connection.cc:535] client connection to 127.0.0.1:34933 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34933): Connection reset by peer (error 104)
2026-09-25T10:44:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1846 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790332740062128-23952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:44:25 dist-test-slave-dxdd krb5kdc[12749](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:44:25 dist-test-slave-dxdd krb5kdc[12749](info): set up 2 sockets
Sep 25 10:44:25 dist-test-slave-dxdd krb5kdc[12749](info): commencing operation
krb5kdc: starting...
W20260925 10:44:26.626775 23952 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Sep 25 10:44:26 dist-test-slave-dxdd krb5kdc[12749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:44:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:44:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790332740062128-23952-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790332740062128-23952-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:44:26.863632 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37433
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790332740062128-23952-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37433
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:44:27.183456 12762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:44:27.183784 12762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:44:27.183954 12762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:44:27.191023 12762 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:44:27.191167 12762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:44:27.191283 12762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:44:27.191387 12762 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:44:27.191471 12762 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:44:27.200829 12762 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790332740062128-23952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37433
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790332740062128-23952-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:44:27.202026 12762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:44:27.203265 12762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:44:27.214490 12767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:27.214576 12768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:27.214604 12770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:44:27.215718 12762 server_base.cc:1034] running on GCE node
I20260925 10:44:27.216368 12762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:44:27.217540 12762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:44:27.218761 12762 hybrid_clock.cc:648] HybridClock initialized: now 1790333067218723 us; error 52 us; skew 500 ppm
I20260925 10:44:27.219106 12762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:44:27 dist-test-slave-dxdd krb5kdc[12749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333067, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:44:27.221977 12762 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:44:27.223384 12762 webserver.cc:460] Webserver started at http://127.0.0.1:36563/ using document root <none> and password file <none>
I20260925 10:44:27.223987 12762 fs_manager.cc:359] Metadata directory not provided
I20260925 10:44:27.224130 12762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:44:27.224411 12762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:44:27.227236 12762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790332740062128-23952-0/minicluster-data/master-0/data/instance:
uuid: "8e7c53d3aa1a47249792749da4b81f5b"
format_stamp: "Formatted at 2026-09-25 10:44:27 on dist-test-slave-dxdd"
server_key: "e53550350b539292da914022370b84a0"
server_key_iv: "92da4fc36ac7266f7434634bd888c595"
server_key_version: "encryptionkey@0"
I20260925 10:44:27.227967 12762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790332740062128-23952-0/minicluster-data/master-0/wal/instance:
uuid: "8e7c53d3aa1a47249792749da4b81f5b"
format_stamp: "Formatted at 2026-09-25 10:44:27 on dist-test-slave-dxdd"
server_key: "e53550350b539292da914022370b84a0"
server_key_iv: "92da4fc36ac7266f7434634bd888c595"
server_key_version: "encryptionkey@0"
I20260925 10:44:27.232769 12762 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260925 10:44:27.235914 12777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:44:27.237170 12762 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 10:44:27.237332 12762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790332740062128-23952-0/minicluster-data/master-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790332740062128-23952-0/minicluster-data/master-0/wal
uuid: "8e7c53d3aa1a47249792749da4b81f5b"
format_stamp: "Formatted at 2026-09-25 10:44:27 on dist-test-slave-dxdd"
server_key: "e53550350b539292da914022370b84a0"
server_key_iv: "92da4fc36ac7266f7434634bd888c595"
server_key_version: "encryptionkey@0"
I20260925 10:44:27.237514 12762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790332740062128-23952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790332740062128-23952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790332740062128-23952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:44:27.243021 12762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:44:27.246630 12762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:44:27.246891 12762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:44:27.264495 12762 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37433
I20260925 10:44:27.264573 12829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37433 every 8 connection(s)
I20260925 10:44:27.265869 12762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
I20260925 10:44:27.267099 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 12762
I20260925 10:44:27.267275 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790332740062128-23952-0/minicluster-data/master-0/wal/instance
I20260925 10:44:27.267805 23952 external_mini_cluster.cc:1428] Setting key cf1f7a1f2179b8b8f0bb6a081d21ae8a
I20260925 10:44:27.270232 12830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:44:27.275184 12833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:27.269518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.183:55803 (local address 127.0.0.1:37433)
0925 10:44:27.270261 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:27.270294 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:27.270365 (+ 71us) server_negotiation.cc:407] Connection header received
0925 10:44:27.271953 (+ 1588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:27.271972 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:27.272293 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:27.272680 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:27.273666 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:27.273671 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:27.273683 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:44:27.273699 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:27.273764 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:44:27.273782 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:27.274020 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:27.274151 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 10:44:27.274355 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":339,"threads_started":1}
I20260925 10:44:27.278693 23952 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:44:27.281405 12830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e7c53d3aa1a47249792749da4b81f5b: Bootstrap starting.
I20260925 10:44:27.284889 12830 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e7c53d3aa1a47249792749da4b81f5b: Neither blocks nor log segments found. Creating new log.
I20260925 10:44:27.285334 12833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:27.283019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.183:56791 (local address 127.0.0.1:37433)
0925 10:44:27.283211 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:27.283218 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:27.283505 (+ 287us) server_negotiation.cc:407] Connection header received
0925 10:44:27.283646 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:27.283649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:27.283690 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:27.283762 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:27.284650 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:27.284654 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:27.284657 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:44:27.284670 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:27.284702 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:44:27.284715 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:27.284800 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:27.284950 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 10:44:27.285021 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 10:44:27.286118 12830 log.cc:824] T 00000000000000000000000000000000 P 8e7c53d3aa1a47249792749da4b81f5b: Log is configured to *not* fsync() on all Append() calls
I20260925 10:44:27.288585 12830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e7c53d3aa1a47249792749da4b81f5b: No bootstrap required, opened a new log
I20260925 10:44:27.295837 12830 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e7c53d3aa1a47249792749da4b81f5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e7c53d3aa1a47249792749da4b81f5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37433 } }
I20260925 10:44:27.296173 12830 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e7c53d3aa1a47249792749da4b81f5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:44:27.296264 12830 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e7c53d3aa1a47249792749da4b81f5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7c53d3aa1a47249792749da4b81f5b, State: Initialized, Role: FOLLOWER
I20260925 10:44:27.296797 12830 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e7c53d3aa1a47249792749da4b81f5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e7c53d3aa1a47249792749da4b81f5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37433 } }
I20260925 10:44:27.297013 12830 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e7c53d3aa1a47249792749da4b81f5b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:44:27.297160 12830 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e7c53d3aa1a47249792749da4b81f5b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:44:27.297314 12830 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e7c53d3aa1a47249792749da4b81f5b [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:44:27.299434 12830 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e7c53d3aa1a47249792749da4b81f5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e7c53d3aa1a47249792749da4b81f5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37433 } }
I20260925 10:44:27.299894 12830 leader_election.cc:302] T 00000000000000000000000000000000 P 8e7c53d3aa1a47249792749da4b81f5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e7c53d3aa1a47249792749da4b81f5b; no voters:
I20260925 10:44:27.300405 12830 leader_election.cc:288] T 00000000000000000000000000000000 P 8e7c53d3aa1a47249792749da4b81f5b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:44:27.300527 12840 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e7c53d3aa1a47249792749da4b81f5b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:44:27.301280 12840 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e7c53d3aa1a47249792749da4b81f5b [term 1 LEADER]: Becoming Leader. State: Replica: 8e7c53d3aa1a47249792749da4b81f5b, State: Running, Role: LEADER
I20260925 10:44:27.301761 12840 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e7c53d3aa1a47249792749da4b81f5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e7c53d3aa1a47249792749da4b81f5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37433 } }
I20260925 10:44:27.302493 12830 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e7c53d3aa1a47249792749da4b81f5b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:44:27.304632 12842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e7c53d3aa1a47249792749da4b81f5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e7c53d3aa1a47249792749da4b81f5b. Latest consensus state: current_term: 1 leader_uuid: "8e7c53d3aa1a47249792749da4b81f5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e7c53d3aa1a47249792749da4b81f5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37433 } } }
I20260925 10:44:27.304613 12841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e7c53d3aa1a47249792749da4b81f5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e7c53d3aa1a47249792749da4b81f5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e7c53d3aa1a47249792749da4b81f5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37433 } } }
I20260925 10:44:27.305038 12842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e7c53d3aa1a47249792749da4b81f5b [sys.catalog]: This master's current role is: LEADER
I20260925 10:44:27.305074 12841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e7c53d3aa1a47249792749da4b81f5b [sys.catalog]: This master's current role is: LEADER
I20260925 10:44:27.307273 12846 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:44:27.323300 12846 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:44:27.331866 12846 catalog_manager.cc:1347] Generated new cluster ID: 5e7f9ea86e614cd3a64f1c50f213dd7f
I20260925 10:44:27.331993 12846 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:44:27.366580 12846 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:44:27.367000 12846 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:44:27.373639 12846 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e7c53d3aa1a47249792749da4b81f5b: Generated new TSK 0
I20260925 10:44:27.374393 12846 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:44:27.379588 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 12762
W20260925 10:44:27.380623 12825 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.183:55803
W20260925 10:44:27.381269 23952 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 12762: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:44:27.381397 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 12762
2026-09-25T10:44:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1851 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790332740062128-23952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:44:27 dist-test-slave-dxdd krb5kdc[12861](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:44:27 dist-test-slave-dxdd krb5kdc[12861](info): set up 2 sockets
Sep 25 10:44:27 dist-test-slave-dxdd krb5kdc[12861](info): commencing operation
krb5kdc: starting...
W20260925 10:44:28.483055 23952 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Sep 25 10:44:28 dist-test-slave-dxdd krb5kdc[12861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:44:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:44:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790332740062128-23952-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790332740062128-23952-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:44:28.731966 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38725
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790332740062128-23952-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:44:29.057657 12874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:44:29.057934 12874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:44:29.058118 12874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:44:29.065400 12874 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:44:29.065500 12874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:44:29.065603 12874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:44:29.065706 12874 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:44:29.065773 12874 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:44:29.075477 12874 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790332740062128-23952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38725
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790332740062128-23952-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:44:29.076745 12874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:44:29.078068 12874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:44:29.089892 12880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:29.090004 12879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:29.089915 12882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:44:29.090576 12874 server_base.cc:1034] running on GCE node
I20260925 10:44:29.091471 12874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:44:29.092870 12874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:44:29.094100 12874 hybrid_clock.cc:648] HybridClock initialized: now 1790333069094061 us; error 53 us; skew 500 ppm
I20260925 10:44:29.094492 12874 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:44:29 dist-test-slave-dxdd krb5kdc[12861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333069, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:44:29.102578 12874 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:44:29.104054 12874 webserver.cc:460] Webserver started at http://127.0.0.1:44941/ using document root <none> and password file <none>
I20260925 10:44:29.104705 12874 fs_manager.cc:359] Metadata directory not provided
I20260925 10:44:29.104801 12874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:44:29.105041 12874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:44:29.107950 12874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790332740062128-23952-0/minicluster-data/master-0/data/instance:
uuid: "94ceacfca4e743bfac0bc4498128e8e1"
format_stamp: "Formatted at 2026-09-25 10:44:29 on dist-test-slave-dxdd"
server_key: "a885d15ee37a7cc7ac87528372648826"
server_key_iv: "449b021e806c3c01528a180bb27fb8e7"
server_key_version: "encryptionkey@0"
I20260925 10:44:29.108647 12874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790332740062128-23952-0/minicluster-data/master-0/wal/instance:
uuid: "94ceacfca4e743bfac0bc4498128e8e1"
format_stamp: "Formatted at 2026-09-25 10:44:29 on dist-test-slave-dxdd"
server_key: "a885d15ee37a7cc7ac87528372648826"
server_key_iv: "449b021e806c3c01528a180bb27fb8e7"
server_key_version: "encryptionkey@0"
I20260925 10:44:29.113507 12874 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 10:44:29.116925 12889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:44:29.118139 12874 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:44:29.118278 12874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790332740062128-23952-0/minicluster-data/master-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790332740062128-23952-0/minicluster-data/master-0/wal
uuid: "94ceacfca4e743bfac0bc4498128e8e1"
format_stamp: "Formatted at 2026-09-25 10:44:29 on dist-test-slave-dxdd"
server_key: "a885d15ee37a7cc7ac87528372648826"
server_key_iv: "449b021e806c3c01528a180bb27fb8e7"
server_key_version: "encryptionkey@0"
I20260925 10:44:29.118492 12874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790332740062128-23952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790332740062128-23952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790332740062128-23952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:44:29.137638 12874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:44:29.141646 12874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:44:29.141880 12874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:44:29.160175 12874 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38725
I20260925 10:44:29.160207 12940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38725 every 8 connection(s)
I20260925 10:44:29.161506 12874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
I20260925 10:44:29.165198 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 12874
I20260925 10:44:29.165381 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790332740062128-23952-0/minicluster-data/master-0/wal/instance
I20260925 10:44:29.165577 12941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:44:29.165948 23952 external_mini_cluster.cc:1428] Setting key 82affb74c95056ed86ad78a9584ea20c
I20260925 10:44:29.179894 12941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94ceacfca4e743bfac0bc4498128e8e1: Bootstrap starting.
Sep 25 10:44:29 dist-test-slave-dxdd krb5kdc[12861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 10:44:29.183223 12941 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94ceacfca4e743bfac0bc4498128e8e1: Neither blocks nor log segments found. Creating new log.
I20260925 10:44:29.184276 12941 log.cc:824] T 00000000000000000000000000000000 P 94ceacfca4e743bfac0bc4498128e8e1: Log is configured to *not* fsync() on all Append() calls
I20260925 10:44:29.186968 12941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94ceacfca4e743bfac0bc4498128e8e1: No bootstrap required, opened a new log
I20260925 10:44:29.188339 12944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:29.167714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.183:49521 (local address 127.0.0.1:38725)
0925 10:44:29.168486 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:29.168497 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:29.168537 (+ 40us) server_negotiation.cc:407] Connection header received
0925 10:44:29.170205 (+ 1668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:29.170226 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:29.170697 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:29.171007 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:29.172677 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:29.173911 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:29.175718 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:29.176366 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:29.176500 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:29.181272 (+ 4772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:29.181283 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:29.181296 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:44:29.181329 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:29.184405 (+ 3076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:29.185164 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:29.185178 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:29.185186 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:29.185364 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:29.185840 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:29.185851 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:29.185854 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:29.186298 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:29.186686 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:29.187149 (+ 463us) server_negotiation.cc:299] Negotiation successful
0925 10:44:29.187452 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":341,"threads_started":1}
I20260925 10:44:29.192030 23952 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:44:29.195010 12941 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94ceacfca4e743bfac0bc4498128e8e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94ceacfca4e743bfac0bc4498128e8e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38725 } }
I20260925 10:44:29.195362 12941 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94ceacfca4e743bfac0bc4498128e8e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:44:29.195492 12941 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94ceacfca4e743bfac0bc4498128e8e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94ceacfca4e743bfac0bc4498128e8e1, State: Initialized, Role: FOLLOWER
I20260925 10:44:29.196040 12941 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94ceacfca4e743bfac0bc4498128e8e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94ceacfca4e743bfac0bc4498128e8e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38725 } }
I20260925 10:44:29.196388 12941 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94ceacfca4e743bfac0bc4498128e8e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:44:29.196553 12941 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94ceacfca4e743bfac0bc4498128e8e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:44:29.196810 12941 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94ceacfca4e743bfac0bc4498128e8e1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:44:29.199048 12941 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94ceacfca4e743bfac0bc4498128e8e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94ceacfca4e743bfac0bc4498128e8e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38725 } }
I20260925 10:44:29.199513 12941 leader_election.cc:302] T 00000000000000000000000000000000 P 94ceacfca4e743bfac0bc4498128e8e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94ceacfca4e743bfac0bc4498128e8e1; no voters:
I20260925 10:44:29.200177 12941 leader_election.cc:288] T 00000000000000000000000000000000 P 94ceacfca4e743bfac0bc4498128e8e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:44:29.200268 12951 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94ceacfca4e743bfac0bc4498128e8e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:44:29.201478 12951 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94ceacfca4e743bfac0bc4498128e8e1 [term 1 LEADER]: Becoming Leader. State: Replica: 94ceacfca4e743bfac0bc4498128e8e1, State: Running, Role: LEADER
I20260925 10:44:29.202121 12951 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94ceacfca4e743bfac0bc4498128e8e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94ceacfca4e743bfac0bc4498128e8e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38725 } }
I20260925 10:44:29.202564 12941 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94ceacfca4e743bfac0bc4498128e8e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:44:29.204923 12953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94ceacfca4e743bfac0bc4498128e8e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94ceacfca4e743bfac0bc4498128e8e1. Latest consensus state: current_term: 1 leader_uuid: "94ceacfca4e743bfac0bc4498128e8e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94ceacfca4e743bfac0bc4498128e8e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38725 } } }
I20260925 10:44:29.205296 12953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94ceacfca4e743bfac0bc4498128e8e1 [sys.catalog]: This master's current role is: LEADER
I20260925 10:44:29.205153 12952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94ceacfca4e743bfac0bc4498128e8e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94ceacfca4e743bfac0bc4498128e8e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94ceacfca4e743bfac0bc4498128e8e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38725 } } }
I20260925 10:44:29.205518 12952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94ceacfca4e743bfac0bc4498128e8e1 [sys.catalog]: This master's current role is: LEADER
I20260925 10:44:29.208227 12959 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:44:29.214788 12944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:29.196456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.183:35287 (local address 127.0.0.1:38725)
0925 10:44:29.196663 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:29.196670 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:29.197059 (+ 389us) server_negotiation.cc:407] Connection header received
0925 10:44:29.197250 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:29.197255 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:29.197323 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:29.197428 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:29.198826 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:29.199800 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:29.201520 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:29.202036 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:29.202146 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:29.205592 (+ 3446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:29.205603 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:29.205611 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:44:29.205652 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:29.210645 (+ 4993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:29.211533 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:29.211542 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:29.211559 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:29.211789 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:29.212400 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:29.212405 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:29.212408 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:29.212777 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:29.213032 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:29.213731 (+ 699us) server_negotiation.cc:299] Negotiation successful
0925 10:44:29.213902 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":72}
I20260925 10:44:29.215730 12959 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:44:29.225728 12959 catalog_manager.cc:1347] Generated new cluster ID: d81b38f16b574236a271799e6c3d3dd5
I20260925 10:44:29.225826 12959 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:44:29.236644 12959 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:44:29.238037 12959 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:44:29.252506 12959 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94ceacfca4e743bfac0bc4498128e8e1: Generated new TSK 0
I20260925 10:44:29.253273 12959 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:44:29.262338 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 12874
I20260925 10:44:29.263553 12936 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:44:29.352329 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 12874
W20260925 10:44:29.369098 12859 connection.cc:535] client connection to 127.0.0.1:38725 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38725): Connection reset by peer (error 104)
2026-09-25T10:44:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1971 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790332740062128-23952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:44:29 dist-test-slave-dxdd krb5kdc[12973](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:44:29 dist-test-slave-dxdd krb5kdc[12973](info): set up 2 sockets
Sep 25 10:44:29 dist-test-slave-dxdd krb5kdc[12973](info): commencing operation
krb5kdc: starting...
W20260925 10:44:30.458889 23952 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Sep 25 10:44:30 dist-test-slave-dxdd krb5kdc[12973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:44:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:44:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790332740062128-23952-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790332740062128-23952-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:44:30.697046 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44271
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790332740062128-23952-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44271
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:44:31.013918 12986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:44:31.014191 12986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:44:31.014384 12986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:44:31.021415 12986 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:44:31.021555 12986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:44:31.021669 12986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:44:31.021770 12986 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:44:31.021854 12986 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:44:31.031675 12986 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790332740062128-23952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44271
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790332740062128-23952-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:44:31.032984 12986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:44:31.034258 12986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:44:31.046069 12994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:31.046391 12992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:44:31.047163 12986 server_base.cc:1034] running on GCE node
W20260925 10:44:31.049543 12991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:44:31.051049 12986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:44:31.052779 12986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:44:31.054016 12986 hybrid_clock.cc:648] HybridClock initialized: now 1790333071053956 us; error 53 us; skew 500 ppm
I20260925 10:44:31.054456 12986 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:44:31 dist-test-slave-dxdd krb5kdc[12973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333071, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:44:31.057431 12986 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:44:31.058920 12986 webserver.cc:460] Webserver started at http://127.0.0.1:44085/ using document root <none> and password file <none>
I20260925 10:44:31.059566 12986 fs_manager.cc:359] Metadata directory not provided
I20260925 10:44:31.059705 12986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:44:31.059988 12986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:44:31.062984 12986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790332740062128-23952-0/minicluster-data/master-0/data/instance:
uuid: "deb66470f36040179c3e2200ad8aada1"
format_stamp: "Formatted at 2026-09-25 10:44:31 on dist-test-slave-dxdd"
server_key: "d73654bcec8a530cc1c84a6019c0a9d3"
server_key_iv: "9809b1bc4c4314abe6ccafe3f26fe6a6"
server_key_version: "encryptionkey@0"
I20260925 10:44:31.063707 12986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790332740062128-23952-0/minicluster-data/master-0/wal/instance:
uuid: "deb66470f36040179c3e2200ad8aada1"
format_stamp: "Formatted at 2026-09-25 10:44:31 on dist-test-slave-dxdd"
server_key: "d73654bcec8a530cc1c84a6019c0a9d3"
server_key_iv: "9809b1bc4c4314abe6ccafe3f26fe6a6"
server_key_version: "encryptionkey@0"
I20260925 10:44:31.068675 12986 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 10:44:31.071940 13001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:44:31.073094 12986 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 10:44:31.073262 12986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790332740062128-23952-0/minicluster-data/master-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790332740062128-23952-0/minicluster-data/master-0/wal
uuid: "deb66470f36040179c3e2200ad8aada1"
format_stamp: "Formatted at 2026-09-25 10:44:31 on dist-test-slave-dxdd"
server_key: "d73654bcec8a530cc1c84a6019c0a9d3"
server_key_iv: "9809b1bc4c4314abe6ccafe3f26fe6a6"
server_key_version: "encryptionkey@0"
I20260925 10:44:31.073438 12986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790332740062128-23952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790332740062128-23952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790332740062128-23952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:44:31.107005 12986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:44:31.110751 12986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:44:31.111025 12986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:44:31.128995 12986 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44271
I20260925 10:44:31.129025 13052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44271 every 8 connection(s)
I20260925 10:44:31.130281 12986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
I20260925 10:44:31.133817 13053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:44:31.140053 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 12986
I20260925 10:44:31.140239 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790332740062128-23952-0/minicluster-data/master-0/wal/instance
I20260925 10:44:31.140648 23952 external_mini_cluster.cc:1428] Setting key fd1c7e96c6a07926ebe2604a33ea83f9
I20260925 10:44:31.144672 13053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P deb66470f36040179c3e2200ad8aada1: Bootstrap starting.
I20260925 10:44:31.148742 13053 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P deb66470f36040179c3e2200ad8aada1: Neither blocks nor log segments found. Creating new log.
I20260925 10:44:31.149866 13053 log.cc:824] T 00000000000000000000000000000000 P deb66470f36040179c3e2200ad8aada1: Log is configured to *not* fsync() on all Append() calls
I20260925 10:44:31.152863 13053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P deb66470f36040179c3e2200ad8aada1: No bootstrap required, opened a new log
I20260925 10:44:31.155640 13056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:31.142543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.183:49463 (local address 127.0.0.1:44271)
0925 10:44:31.143260 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:31.143276 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:31.143346 (+ 70us) server_negotiation.cc:407] Connection header received
0925 10:44:31.145714 (+ 2368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:31.145741 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:31.146218 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:31.146725 (+ 507us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:31.147237 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:31.149288 (+ 2051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:31.150808 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:31.151719 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:31.151903 (+ 184us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:31.152997 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:31.153008 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:31.153025 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:44:31.153728 (+ 703us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:31.153811 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:44:31.153842 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:31.154006 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:31.154238 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 10:44:31.154416 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":495,"thread_start_us":295,"threads_started":1}
I20260925 10:44:31.159425 23952 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:44:31.161972 13053 raft_consensus.cc:348] T 00000000000000000000000000000000 P deb66470f36040179c3e2200ad8aada1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deb66470f36040179c3e2200ad8aada1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44271 } }
I20260925 10:44:31.162508 13053 raft_consensus.cc:374] T 00000000000000000000000000000000 P deb66470f36040179c3e2200ad8aada1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:44:31.162632 13053 raft_consensus.cc:729] T 00000000000000000000000000000000 P deb66470f36040179c3e2200ad8aada1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: deb66470f36040179c3e2200ad8aada1, State: Initialized, Role: FOLLOWER
I20260925 10:44:31.163239 13053 consensus_queue.cc:260] T 00000000000000000000000000000000 P deb66470f36040179c3e2200ad8aada1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deb66470f36040179c3e2200ad8aada1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44271 } }
I20260925 10:44:31.163499 13053 raft_consensus.cc:388] T 00000000000000000000000000000000 P deb66470f36040179c3e2200ad8aada1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:44:31.163666 13053 raft_consensus.cc:482] T 00000000000000000000000000000000 P deb66470f36040179c3e2200ad8aada1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:44:31.163897 13053 raft_consensus.cc:3037] T 00000000000000000000000000000000 P deb66470f36040179c3e2200ad8aada1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:44:31.166865 13053 raft_consensus.cc:504] T 00000000000000000000000000000000 P deb66470f36040179c3e2200ad8aada1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deb66470f36040179c3e2200ad8aada1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44271 } }
I20260925 10:44:31.167416 13053 leader_election.cc:302] T 00000000000000000000000000000000 P deb66470f36040179c3e2200ad8aada1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: deb66470f36040179c3e2200ad8aada1; no voters:
I20260925 10:44:31.168057 13053 leader_election.cc:288] T 00000000000000000000000000000000 P deb66470f36040179c3e2200ad8aada1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:44:31.168164 13063 raft_consensus.cc:2781] T 00000000000000000000000000000000 P deb66470f36040179c3e2200ad8aada1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:44:31.170372 13063 raft_consensus.cc:686] T 00000000000000000000000000000000 P deb66470f36040179c3e2200ad8aada1 [term 1 LEADER]: Becoming Leader. State: Replica: deb66470f36040179c3e2200ad8aada1, State: Running, Role: LEADER
I20260925 10:44:31.170863 13053 sys_catalog.cc:564] T 00000000000000000000000000000000 P deb66470f36040179c3e2200ad8aada1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:44:31.170918 13063 consensus_queue.cc:237] T 00000000000000000000000000000000 P deb66470f36040179c3e2200ad8aada1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deb66470f36040179c3e2200ad8aada1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44271 } }
I20260925 10:44:31.171876 13056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:31.163821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.183:45341 (local address 127.0.0.1:44271)
0925 10:44:31.164082 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:31.164091 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:31.164558 (+ 467us) server_negotiation.cc:407] Connection header received
0925 10:44:31.164737 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:31.164743 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:31.164825 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:31.164893 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:31.165580 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:31.166296 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:31.168239 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:31.169648 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:31.169763 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:31.171057 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:31.171063 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:31.171069 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:44:31.171092 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:31.171155 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:44:31.171178 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:31.171287 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:31.171484 (+ 197us) server_negotiation.cc:299] Negotiation successful
0925 10:44:31.171544 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 10:44:31.174376 13064 sys_catalog.cc:455] T 00000000000000000000000000000000 P deb66470f36040179c3e2200ad8aada1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "deb66470f36040179c3e2200ad8aada1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deb66470f36040179c3e2200ad8aada1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44271 } } }
I20260925 10:44:31.174355 13065 sys_catalog.cc:455] T 00000000000000000000000000000000 P deb66470f36040179c3e2200ad8aada1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader deb66470f36040179c3e2200ad8aada1. Latest consensus state: current_term: 1 leader_uuid: "deb66470f36040179c3e2200ad8aada1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deb66470f36040179c3e2200ad8aada1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44271 } } }
I20260925 10:44:31.174799 13064 sys_catalog.cc:458] T 00000000000000000000000000000000 P deb66470f36040179c3e2200ad8aada1 [sys.catalog]: This master's current role is: LEADER
I20260925 10:44:31.174826 13065 sys_catalog.cc:458] T 00000000000000000000000000000000 P deb66470f36040179c3e2200ad8aada1 [sys.catalog]: This master's current role is: LEADER
I20260925 10:44:31.179359 13069 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:44:31.185007 13069 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:44:31.193094 13069 catalog_manager.cc:1347] Generated new cluster ID: 95fd8eba2d4c4bb8b6a9adce79d83a0a
I20260925 10:44:31.193197 13069 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:44:31.207404 13069 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:44:31.209074 13069 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:44:31.220130 13069 catalog_manager.cc:5868] T 00000000000000000000000000000000 P deb66470f36040179c3e2200ad8aada1: Generated new TSK 0
I20260925 10:44:31.221074 13069 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:44:31.227389 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 12986
W20260925 10:44:31.228446 13048 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.183:49463
W20260925 10:44:31.229174 23952 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 12986: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:44:31.229326 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 12986
W20260925 10:44:31.245901 12971 connection.cc:535] client connection to 127.0.0.1:44271 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44271): Connection reset by peer (error 104)
2026-09-25T10:44:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1876 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790332740062128-23952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:44:31 dist-test-slave-dxdd krb5kdc[13084](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:44:31 dist-test-slave-dxdd krb5kdc[13084](info): set up 2 sockets
Sep 25 10:44:31 dist-test-slave-dxdd krb5kdc[13084](info): commencing operation
krb5kdc: starting...
W20260925 10:44:32.298938 23952 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Sep 25 10:44:32 dist-test-slave-dxdd krb5kdc[13084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:44:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:44:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790332740062128-23952-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790332740062128-23952-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:44:32.535578 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37531
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790332740062128-23952-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37531
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:44:32.849555 13097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:44:32.849872 13097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:44:32.850068 13097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:44:32.857295 13097 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:44:32.857450 13097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:44:32.857542 13097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:44:32.857630 13097 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:44:32.857697 13097 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:44:32.867861 13097 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790332740062128-23952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37531
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790332740062128-23952-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:44:32.869215 13097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:44:32.870535 13097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:44:32.882004 13102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:32.882050 13103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:32.882004 13105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:44:32.883155 13097 server_base.cc:1034] running on GCE node
I20260925 10:44:32.883765 13097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:44:32.884933 13097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:44:32.886132 13097 hybrid_clock.cc:648] HybridClock initialized: now 1790333072886094 us; error 30 us; skew 500 ppm
I20260925 10:44:32.886480 13097 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:44:32 dist-test-slave-dxdd krb5kdc[13084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333072, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:44:32.889278 13097 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:44:32.890618 13097 webserver.cc:460] Webserver started at http://127.0.0.1:33725/ using document root <none> and password file <none>
I20260925 10:44:32.891209 13097 fs_manager.cc:359] Metadata directory not provided
I20260925 10:44:32.891302 13097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:44:32.891541 13097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:44:32.894279 13097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790332740062128-23952-0/minicluster-data/master-0/data/instance:
uuid: "38b161bd303d4c64ac919dc515c6b498"
format_stamp: "Formatted at 2026-09-25 10:44:32 on dist-test-slave-dxdd"
server_key: "ba9b76f11157af0f8dd6393c3862aff8"
server_key_iv: "3c38c0969cfa004f92bd4df2edf85241"
server_key_version: "encryptionkey@0"
I20260925 10:44:32.895009 13097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790332740062128-23952-0/minicluster-data/master-0/wal/instance:
uuid: "38b161bd303d4c64ac919dc515c6b498"
format_stamp: "Formatted at 2026-09-25 10:44:32 on dist-test-slave-dxdd"
server_key: "ba9b76f11157af0f8dd6393c3862aff8"
server_key_iv: "3c38c0969cfa004f92bd4df2edf85241"
server_key_version: "encryptionkey@0"
I20260925 10:44:32.899766 13097 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 10:44:32.902905 13112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:44:32.904119 13097 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:44:32.904241 13097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790332740062128-23952-0/minicluster-data/master-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790332740062128-23952-0/minicluster-data/master-0/wal
uuid: "38b161bd303d4c64ac919dc515c6b498"
format_stamp: "Formatted at 2026-09-25 10:44:32 on dist-test-slave-dxdd"
server_key: "ba9b76f11157af0f8dd6393c3862aff8"
server_key_iv: "3c38c0969cfa004f92bd4df2edf85241"
server_key_version: "encryptionkey@0"
I20260925 10:44:32.904390 13097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790332740062128-23952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790332740062128-23952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790332740062128-23952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:44:32.909758 13097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:44:32.913808 13097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:44:32.914038 13097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:44:32.930672 13097 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37531
I20260925 10:44:32.930750 13163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37531 every 8 connection(s)
I20260925 10:44:32.931967 13097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
I20260925 10:44:32.935483 13164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:44:32.936604 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 13097
I20260925 10:44:32.936764 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790332740062128-23952-0/minicluster-data/master-0/wal/instance
I20260925 10:44:32.937175 23952 external_mini_cluster.cc:1428] Setting key 90b15cdb3b7d8525a7fc1316124885d2
I20260925 10:44:32.945513 13167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:32.938975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.183:46087 (local address 127.0.0.1:37531)
0925 10:44:32.939731 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:32.939745 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:32.939793 (+ 48us) server_negotiation.cc:407] Connection header received
0925 10:44:32.941883 (+ 2090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:32.941911 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:32.942352 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:32.942864 (+ 512us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:32.943747 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:32.943755 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:32.943772 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:44:32.943786 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:32.943845 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:44:32.943871 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:32.944068 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:32.944197 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 10:44:32.944370 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":284,"threads_started":1}
I20260925 10:44:32.949051 23952 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:44:32.949605 13164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38b161bd303d4c64ac919dc515c6b498: Bootstrap starting.
I20260925 10:44:32.952404 13164 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38b161bd303d4c64ac919dc515c6b498: Neither blocks nor log segments found. Creating new log.
I20260925 10:44:32.953366 13164 log.cc:824] T 00000000000000000000000000000000 P 38b161bd303d4c64ac919dc515c6b498: Log is configured to *not* fsync() on all Append() calls
I20260925 10:44:32.955932 13164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38b161bd303d4c64ac919dc515c6b498: No bootstrap required, opened a new log
I20260925 10:44:32.959837 13167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:32.957438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.183:46045 (local address 127.0.0.1:37531)
0925 10:44:32.957600 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:32.957606 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:32.958024 (+ 418us) server_negotiation.cc:407] Connection header received
0925 10:44:32.958139 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:32.958144 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:32.958217 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:32.958335 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:32.959227 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:32.959233 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:32.959238 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:44:32.959254 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:32.959295 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:44:32.959314 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:32.959453 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:32.959512 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 10:44:32.959569 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 10:44:32.964128 13164 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38b161bd303d4c64ac919dc515c6b498 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b161bd303d4c64ac919dc515c6b498" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37531 } }
I20260925 10:44:32.964460 13164 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38b161bd303d4c64ac919dc515c6b498 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:44:32.964576 13164 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38b161bd303d4c64ac919dc515c6b498 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38b161bd303d4c64ac919dc515c6b498, State: Initialized, Role: FOLLOWER
I20260925 10:44:32.965229 13164 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38b161bd303d4c64ac919dc515c6b498 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b161bd303d4c64ac919dc515c6b498" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37531 } }
I20260925 10:44:32.965520 13164 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38b161bd303d4c64ac919dc515c6b498 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:44:32.965662 13164 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38b161bd303d4c64ac919dc515c6b498 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:44:32.965821 13164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38b161bd303d4c64ac919dc515c6b498 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:44:32.968140 13164 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38b161bd303d4c64ac919dc515c6b498 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b161bd303d4c64ac919dc515c6b498" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37531 } }
I20260925 10:44:32.968607 13164 leader_election.cc:302] T 00000000000000000000000000000000 P 38b161bd303d4c64ac919dc515c6b498 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38b161bd303d4c64ac919dc515c6b498; no voters:
I20260925 10:44:32.969215 13164 leader_election.cc:288] T 00000000000000000000000000000000 P 38b161bd303d4c64ac919dc515c6b498 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:44:32.969372 13174 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38b161bd303d4c64ac919dc515c6b498 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:44:32.970077 13174 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38b161bd303d4c64ac919dc515c6b498 [term 1 LEADER]: Becoming Leader. State: Replica: 38b161bd303d4c64ac919dc515c6b498, State: Running, Role: LEADER
I20260925 10:44:32.970602 13174 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38b161bd303d4c64ac919dc515c6b498 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b161bd303d4c64ac919dc515c6b498" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37531 } }
I20260925 10:44:32.971297 13164 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38b161bd303d4c64ac919dc515c6b498 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:44:32.973424 13176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38b161bd303d4c64ac919dc515c6b498 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38b161bd303d4c64ac919dc515c6b498. Latest consensus state: current_term: 1 leader_uuid: "38b161bd303d4c64ac919dc515c6b498" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b161bd303d4c64ac919dc515c6b498" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37531 } } }
I20260925 10:44:32.973896 13176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38b161bd303d4c64ac919dc515c6b498 [sys.catalog]: This master's current role is: LEADER
I20260925 10:44:32.974448 13175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38b161bd303d4c64ac919dc515c6b498 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38b161bd303d4c64ac919dc515c6b498" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b161bd303d4c64ac919dc515c6b498" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37531 } } }
I20260925 10:44:32.974781 13175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38b161bd303d4c64ac919dc515c6b498 [sys.catalog]: This master's current role is: LEADER
I20260925 10:44:32.975529 13182 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:44:32.980057 13182 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:44:32.988569 13182 catalog_manager.cc:1347] Generated new cluster ID: c2c8a39773574d5aa2cd8f240ab1b2f9
I20260925 10:44:32.988677 13182 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:44:33.005543 13182 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:44:33.006042 13182 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:44:33.012245 13182 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38b161bd303d4c64ac919dc515c6b498: Generated new TSK 0
I20260925 10:44:33.012943 13182 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:44:33.023707 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 13097
W20260925 10:44:33.024706 13159 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.183:46087
W20260925 10:44:33.025334 23952 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 13097: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:44:33.025476 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 13097
2026-09-25T10:44:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1796 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790332740062128-23952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:44:33 dist-test-slave-dxdd krb5kdc[13195](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:44:33 dist-test-slave-dxdd krb5kdc[13195](info): set up 2 sockets
Sep 25 10:44:33 dist-test-slave-dxdd krb5kdc[13195](info): commencing operation
krb5kdc: starting...
W20260925 10:44:34.106717 23952 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Sep 25 10:44:34 dist-test-slave-dxdd krb5kdc[13195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:44:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:44:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
I20260925 10:44:34.346988 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:43333
--webserver_interface=127.23.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:42119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.62:43333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:44:34.668427 13208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:44:34.668689 13208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:44:34.668848 13208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:44:34.676014 13208 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:44:34.676117 13208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:44:34.676229 13208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:44:34.676357 13208 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:44:34.676435 13208 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:44:34.685858 13208 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:42119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790332740062128-23952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.62:43333
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:43333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:44:34.687104 13208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:44:34.688386 13208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:44:34.699717 13216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:34.699776 13213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:34.699862 13214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:44:34.700529 13208 server_base.cc:1034] running on GCE node
I20260925 10:44:34.701195 13208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:44:34.702497 13208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:44:34.703693 13208 hybrid_clock.cc:648] HybridClock initialized: now 1790333074703649 us; error 35 us; skew 500 ppm
I20260925 10:44:34.704034 13208 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:44:34 dist-test-slave-dxdd krb5kdc[13195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333074, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:44:34.711493 13208 init.cc:377] Logged in from keytab as kudu/127.23.100.62@KRBTEST.COM (short username kudu)
I20260925 10:44:34.712859 13208 webserver.cc:460] Webserver started at http://127.23.100.62:36917/ using document root <none> and password file <none>
I20260925 10:44:34.713511 13208 fs_manager.cc:359] Metadata directory not provided
I20260925 10:44:34.713658 13208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:44:34.713945 13208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:44:34.716742 13208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790332740062128-23952-0/minicluster-data/master-0/data/instance:
uuid: "e2df6c98d93546bcbf9843050a97124f"
format_stamp: "Formatted at 2026-09-25 10:44:34 on dist-test-slave-dxdd"
server_key: "b53fa4c5fa0ff7868118ad259a8603a8"
server_key_iv: "f4f313ce2248936b00c777a82c4c7d56"
server_key_version: "encryptionkey@0"
I20260925 10:44:34.717445 13208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790332740062128-23952-0/minicluster-data/master-0/wal/instance:
uuid: "e2df6c98d93546bcbf9843050a97124f"
format_stamp: "Formatted at 2026-09-25 10:44:34 on dist-test-slave-dxdd"
server_key: "b53fa4c5fa0ff7868118ad259a8603a8"
server_key_iv: "f4f313ce2248936b00c777a82c4c7d56"
server_key_version: "encryptionkey@0"
I20260925 10:44:34.722383 13208 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 10:44:34.725513 13223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:44:34.726799 13208 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 10:44:34.726969 13208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790332740062128-23952-0/minicluster-data/master-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790332740062128-23952-0/minicluster-data/master-0/wal
uuid: "e2df6c98d93546bcbf9843050a97124f"
format_stamp: "Formatted at 2026-09-25 10:44:34 on dist-test-slave-dxdd"
server_key: "b53fa4c5fa0ff7868118ad259a8603a8"
server_key_iv: "f4f313ce2248936b00c777a82c4c7d56"
server_key_version: "encryptionkey@0"
I20260925 10:44:34.727169 13208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790332740062128-23952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790332740062128-23952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790332740062128-23952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:44:34.749711 13208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:44:34.753935 13208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:44:34.754225 13208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:44:34.771786 13208 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.62:43333
I20260925 10:44:34.771786 13274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.62:43333 every 8 connection(s)
I20260925 10:44:34.773113 13208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
I20260925 10:44:34.776574 13275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:44:34.781870 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 13208
I20260925 10:44:34.782058 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790332740062128-23952-0/minicluster-data/master-0/wal/instance
I20260925 10:44:34.782635 23952 external_mini_cluster.cc:1428] Setting key 9f158eefd025ddacab32870fb0ac2982
I20260925 10:44:34.787503 13275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2df6c98d93546bcbf9843050a97124f: Bootstrap starting.
I20260925 10:44:34.790388 13275 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2df6c98d93546bcbf9843050a97124f: Neither blocks nor log segments found. Creating new log.
I20260925 10:44:34.791339 13275 log.cc:824] T 00000000000000000000000000000000 P e2df6c98d93546bcbf9843050a97124f: Log is configured to *not* fsync() on all Append() calls
I20260925 10:44:34.793944 13275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2df6c98d93546bcbf9843050a97124f: No bootstrap required, opened a new log
Sep 25 10:44:34 dist-test-slave-dxdd krb5kdc[13195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:44:34.802700 13275 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2df6c98d93546bcbf9843050a97124f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2df6c98d93546bcbf9843050a97124f" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 43333 } }
I20260925 10:44:34.803148 13275 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2df6c98d93546bcbf9843050a97124f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:44:34.803268 13275 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2df6c98d93546bcbf9843050a97124f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2df6c98d93546bcbf9843050a97124f, State: Initialized, Role: FOLLOWER
I20260925 10:44:34.803903 13275 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2df6c98d93546bcbf9843050a97124f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2df6c98d93546bcbf9843050a97124f" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 43333 } }
I20260925 10:44:34.804163 13275 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2df6c98d93546bcbf9843050a97124f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:44:34.804314 13275 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2df6c98d93546bcbf9843050a97124f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:44:34.804491 13275 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2df6c98d93546bcbf9843050a97124f [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:44:34.804584 13278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:34.784489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45273 (local address 127.23.100.62:43333)
0925 10:44:34.785278 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:34.785288 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:34.785328 (+ 40us) server_negotiation.cc:407] Connection header received
0925 10:44:34.786987 (+ 1659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:34.787008 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:34.787386 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:34.787739 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:34.789164 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:34.790471 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:34.791755 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:34.792467 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:34.792653 (+ 186us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:34.797186 (+ 4533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:34.797193 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:34.797209 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:44:34.797240 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:34.800337 (+ 3097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:34.801218 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:34.801224 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:34.801229 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:34.801382 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:34.801844 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:34.801847 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:34.801849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:34.802282 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:34.802552 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:34.803097 (+ 545us) server_negotiation.cc:299] Negotiation successful
0925 10:44:34.803470 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":356,"threads_started":1}
I20260925 10:44:34.806852 13275 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2df6c98d93546bcbf9843050a97124f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2df6c98d93546bcbf9843050a97124f" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 43333 } }
I20260925 10:44:34.807408 13275 leader_election.cc:302] T 00000000000000000000000000000000 P e2df6c98d93546bcbf9843050a97124f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2df6c98d93546bcbf9843050a97124f; no voters:
I20260925 10:44:34.808081 13275 leader_election.cc:288] T 00000000000000000000000000000000 P e2df6c98d93546bcbf9843050a97124f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:44:34.808213 23952 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:44:34.808202 13280 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2df6c98d93546bcbf9843050a97124f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:44:34.810266 13275 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2df6c98d93546bcbf9843050a97124f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:44:34.815040 13280 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2df6c98d93546bcbf9843050a97124f [term 1 LEADER]: Becoming Leader. State: Replica: e2df6c98d93546bcbf9843050a97124f, State: Running, Role: LEADER
I20260925 10:44:34.815490 13280 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2df6c98d93546bcbf9843050a97124f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2df6c98d93546bcbf9843050a97124f" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 43333 } }
I20260925 10:44:34.818513 13281 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2df6c98d93546bcbf9843050a97124f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2df6c98d93546bcbf9843050a97124f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2df6c98d93546bcbf9843050a97124f" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 43333 } } }
I20260925 10:44:34.818497 13282 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2df6c98d93546bcbf9843050a97124f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2df6c98d93546bcbf9843050a97124f. Latest consensus state: current_term: 1 leader_uuid: "e2df6c98d93546bcbf9843050a97124f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2df6c98d93546bcbf9843050a97124f" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 43333 } } }
I20260925 10:44:34.818933 13281 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2df6c98d93546bcbf9843050a97124f [sys.catalog]: This master's current role is: LEADER
I20260925 10:44:34.818933 13282 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2df6c98d93546bcbf9843050a97124f [sys.catalog]: This master's current role is: LEADER
I20260925 10:44:34.819761 13289 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:44:34.836625 13289 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:44:34.849416 13289 catalog_manager.cc:1347] Generated new cluster ID: c86ade3cb5c141baa098dfcbc4f85306
I20260925 10:44:34.849539 13289 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:44:34.867231 13278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:34.844665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34143 (local address 127.23.100.62:43333)
0925 10:44:34.845132 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:34.845139 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:34.845400 (+ 261us) server_negotiation.cc:407] Connection header received
0925 10:44:34.845628 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:34.845637 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:34.845738 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:34.845853 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:34.848203 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:34.849968 (+ 1765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:34.852431 (+ 2463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:34.853379 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:34.853502 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:34.857805 (+ 4303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:34.857815 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:34.857823 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:44:34.857870 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:34.862479 (+ 4609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:34.863743 (+ 1264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:34.863752 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:34.863758 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:34.863992 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:34.864706 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:34.864714 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:34.864719 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:34.865250 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:34.865603 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:34.866443 (+ 840us) server_negotiation.cc:299] Negotiation successful
0925 10:44:34.866689 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351}
I20260925 10:44:34.874709 13289 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:44:34.874742 13303 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 10:44:34.876771 13289 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:44:34.884143 13289 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2df6c98d93546bcbf9843050a97124f: Generated new TSK 0
I20260925 10:44:34.885138 13289 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:44:34.893458 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 13208
I20260925 10:44:34.894882 13270 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:44:34.984226 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 13208
W20260925 10:44:35.001385 13193 connection.cc:535] client connection to 127.23.100.62:43333 recv error: Network error: failed to read from TLS socket (remote: 127.23.100.62:43333): Connection reset by peer (error 104)
2026-09-25T10:44:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1959 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790332740062128-23952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:44:35 dist-test-slave-dxdd krb5kdc[13307](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:44:35 dist-test-slave-dxdd krb5kdc[13307](info): set up 2 sockets
Sep 25 10:44:35 dist-test-slave-dxdd krb5kdc[13307](info): commencing operation
krb5kdc: starting...
W20260925 10:44:36.094796 23952 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.003s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Sep 25 10:44:36 dist-test-slave-dxdd krb5kdc[13307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:44:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:44:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
I20260925 10:44:36.338613 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:33401
--webserver_interface=127.23.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:44613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.62:33401
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:44:36.657037 13320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:44:36.657327 13320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:44:36.657492 13320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:44:36.664588 13320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:44:36.664707 13320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:44:36.664805 13320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:44:36.664904 13320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:44:36.664983 13320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:44:36.674510 13320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:44613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790332740062128-23952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.62:33401
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:33401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:44:36.675701 13320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:44:36.677018 13320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:44:36.688562 13325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:36.688637 13329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:36.688613 13327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:44:36.689406 13320 server_base.cc:1034] running on GCE node
I20260925 10:44:36.690132 13320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:44:36.691444 13320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:44:36.692644 13320 hybrid_clock.cc:648] HybridClock initialized: now 1790333076692596 us; error 40 us; skew 500 ppm
I20260925 10:44:36.692998 13320 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:44:36 dist-test-slave-dxdd krb5kdc[13307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333076, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:44:36.696246 13320 init.cc:377] Logged in from keytab as kudu/127.23.100.62@KRBTEST.COM (short username kudu)
I20260925 10:44:36.697610 13320 webserver.cc:460] Webserver started at http://127.23.100.62:43279/ using document root <none> and password file <none>
I20260925 10:44:36.698274 13320 fs_manager.cc:359] Metadata directory not provided
I20260925 10:44:36.698472 13320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:44:36.698766 13320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:44:36.701664 13320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790332740062128-23952-0/minicluster-data/master-0/data/instance:
uuid: "f06603ab6d254cdd8bc38ad304391295"
format_stamp: "Formatted at 2026-09-25 10:44:36 on dist-test-slave-dxdd"
server_key: "dbe12b4a5eaa842e590c11a6863c13ad"
server_key_iv: "22fbcb5684a065c3378173d9f074d6ab"
server_key_version: "encryptionkey@0"
I20260925 10:44:36.702455 13320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790332740062128-23952-0/minicluster-data/master-0/wal/instance:
uuid: "f06603ab6d254cdd8bc38ad304391295"
format_stamp: "Formatted at 2026-09-25 10:44:36 on dist-test-slave-dxdd"
server_key: "dbe12b4a5eaa842e590c11a6863c13ad"
server_key_iv: "22fbcb5684a065c3378173d9f074d6ab"
server_key_version: "encryptionkey@0"
I20260925 10:44:36.707275 13320 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 10:44:36.710500 13336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:44:36.711680 13320 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 10:44:36.711863 13320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790332740062128-23952-0/minicluster-data/master-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790332740062128-23952-0/minicluster-data/master-0/wal
uuid: "f06603ab6d254cdd8bc38ad304391295"
format_stamp: "Formatted at 2026-09-25 10:44:36 on dist-test-slave-dxdd"
server_key: "dbe12b4a5eaa842e590c11a6863c13ad"
server_key_iv: "22fbcb5684a065c3378173d9f074d6ab"
server_key_version: "encryptionkey@0"
I20260925 10:44:36.712050 13320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790332740062128-23952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790332740062128-23952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790332740062128-23952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:44:36.728439 13320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:44:36.732118 13320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:44:36.732424 13320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:44:36.750167 13320 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.62:33401
I20260925 10:44:36.750187 13387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.62:33401 every 8 connection(s)
I20260925 10:44:36.751475 13320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
I20260925 10:44:36.754827 13388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:44:36.760885 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 13320
I20260925 10:44:36.761083 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790332740062128-23952-0/minicluster-data/master-0/wal/instance
I20260925 10:44:36.761494 23952 external_mini_cluster.cc:1428] Setting key f1cb01607480ae0473263b8cac163987
I20260925 10:44:36.765340 13388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f06603ab6d254cdd8bc38ad304391295: Bootstrap starting.
I20260925 10:44:36.769398 13388 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f06603ab6d254cdd8bc38ad304391295: Neither blocks nor log segments found. Creating new log.
I20260925 10:44:36.770610 13388 log.cc:824] T 00000000000000000000000000000000 P f06603ab6d254cdd8bc38ad304391295: Log is configured to *not* fsync() on all Append() calls
I20260925 10:44:36.773890 13388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f06603ab6d254cdd8bc38ad304391295: No bootstrap required, opened a new log
I20260925 10:44:36.776713 13391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:36.763350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59311 (local address 127.23.100.62:33401)
0925 10:44:36.764126 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:36.764140 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:36.764186 (+ 46us) server_negotiation.cc:407] Connection header received
0925 10:44:36.766531 (+ 2345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:36.766556 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:36.767044 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:36.767465 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:36.768056 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:36.770053 (+ 1997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:36.771628 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:36.772569 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:36.772756 (+ 187us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:36.773904 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:36.773918 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:36.773938 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:44:36.774669 (+ 731us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:36.774774 (+ 105us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:44:36.774806 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:36.774960 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:36.775280 (+ 320us) server_negotiation.cc:299] Negotiation successful
0925 10:44:36.775461 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":349,"threads_started":1}
I20260925 10:44:36.780629 23952 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:44:36.782959 13388 raft_consensus.cc:348] T 00000000000000000000000000000000 P f06603ab6d254cdd8bc38ad304391295 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f06603ab6d254cdd8bc38ad304391295" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33401 } }
I20260925 10:44:36.783402 13388 raft_consensus.cc:374] T 00000000000000000000000000000000 P f06603ab6d254cdd8bc38ad304391295 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:44:36.783524 13388 raft_consensus.cc:729] T 00000000000000000000000000000000 P f06603ab6d254cdd8bc38ad304391295 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f06603ab6d254cdd8bc38ad304391295, State: Initialized, Role: FOLLOWER
I20260925 10:44:36.784142 13388 consensus_queue.cc:260] T 00000000000000000000000000000000 P f06603ab6d254cdd8bc38ad304391295 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f06603ab6d254cdd8bc38ad304391295" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33401 } }
I20260925 10:44:36.784376 13388 raft_consensus.cc:388] T 00000000000000000000000000000000 P f06603ab6d254cdd8bc38ad304391295 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:44:36.784539 13388 raft_consensus.cc:482] T 00000000000000000000000000000000 P f06603ab6d254cdd8bc38ad304391295 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:44:36.784706 13388 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f06603ab6d254cdd8bc38ad304391295 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:44:36.788015 13388 raft_consensus.cc:504] T 00000000000000000000000000000000 P f06603ab6d254cdd8bc38ad304391295 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f06603ab6d254cdd8bc38ad304391295" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33401 } }
I20260925 10:44:36.788476 13388 leader_election.cc:302] T 00000000000000000000000000000000 P f06603ab6d254cdd8bc38ad304391295 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f06603ab6d254cdd8bc38ad304391295; no voters:
I20260925 10:44:36.789090 13388 leader_election.cc:288] T 00000000000000000000000000000000 P f06603ab6d254cdd8bc38ad304391295 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:44:36.789211 13398 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f06603ab6d254cdd8bc38ad304391295 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:44:36.791343 13388 sys_catalog.cc:564] T 00000000000000000000000000000000 P f06603ab6d254cdd8bc38ad304391295 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:44:36.792622 13391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:36.785061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46175 (local address 127.23.100.62:33401)
0925 10:44:36.785214 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:36.785217 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:36.785672 (+ 455us) server_negotiation.cc:407] Connection header received
0925 10:44:36.785807 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:36.785812 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:36.785889 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:36.785986 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:36.786475 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:36.787605 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:36.789047 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:36.789852 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:36.789970 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:36.791549 (+ 1579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:36.791556 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:36.791560 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:44:36.791580 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:36.791622 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:44:36.791642 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:36.791744 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:36.792191 (+ 447us) server_negotiation.cc:299] Negotiation successful
0925 10:44:36.792242 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 10:44:36.795143 13398 raft_consensus.cc:686] T 00000000000000000000000000000000 P f06603ab6d254cdd8bc38ad304391295 [term 1 LEADER]: Becoming Leader. State: Replica: f06603ab6d254cdd8bc38ad304391295, State: Running, Role: LEADER
I20260925 10:44:36.795657 13398 consensus_queue.cc:237] T 00000000000000000000000000000000 P f06603ab6d254cdd8bc38ad304391295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f06603ab6d254cdd8bc38ad304391295" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33401 } }
I20260925 10:44:36.798399 13399 sys_catalog.cc:455] T 00000000000000000000000000000000 P f06603ab6d254cdd8bc38ad304391295 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f06603ab6d254cdd8bc38ad304391295" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f06603ab6d254cdd8bc38ad304391295" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33401 } } }
I20260925 10:44:36.798501 13400 sys_catalog.cc:455] T 00000000000000000000000000000000 P f06603ab6d254cdd8bc38ad304391295 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f06603ab6d254cdd8bc38ad304391295. Latest consensus state: current_term: 1 leader_uuid: "f06603ab6d254cdd8bc38ad304391295" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f06603ab6d254cdd8bc38ad304391295" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33401 } } }
I20260925 10:44:36.798873 13399 sys_catalog.cc:458] T 00000000000000000000000000000000 P f06603ab6d254cdd8bc38ad304391295 [sys.catalog]: This master's current role is: LEADER
I20260925 10:44:36.798892 13400 sys_catalog.cc:458] T 00000000000000000000000000000000 P f06603ab6d254cdd8bc38ad304391295 [sys.catalog]: This master's current role is: LEADER
I20260925 10:44:36.799821 13406 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:44:36.805774 13406 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:44:36.813983 13406 catalog_manager.cc:1347] Generated new cluster ID: 96d9073ac6774b71b269d49376964350
I20260925 10:44:36.814087 13406 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:44:36.841449 13406 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:44:36.842842 13406 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:44:36.852792 13406 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f06603ab6d254cdd8bc38ad304391295: Generated new TSK 0
I20260925 10:44:36.853585 13406 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:44:36.863998 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 13320
W20260925 10:44:36.865284 13383 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59311
W20260925 10:44:36.866026 23952 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 13320: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:44:36.866168 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 13320
W20260925 10:44:36.882781 13305 connection.cc:535] client connection to 127.23.100.62:33401 recv error: Network error: failed to read from TLS socket (remote: 127.23.100.62:33401): Connection reset by peer (error 104)
2026-09-25T10:44:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1881 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790332740062128-23952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:44:36 dist-test-slave-dxdd krb5kdc[13419](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:44:36 dist-test-slave-dxdd krb5kdc[13419](info): set up 2 sockets
Sep 25 10:44:36 dist-test-slave-dxdd krb5kdc[13419](info): commencing operation
krb5kdc: starting...
W20260925 10:44:37.974864 23952 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.006s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Sep 25 10:44:38 dist-test-slave-dxdd krb5kdc[13419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:44:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:44:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
I20260925 10:44:38.222952 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:41681
--webserver_interface=127.23.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:46425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.62:41681
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:44:38.546605 13432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:44:38.546860 13432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:44:38.547046 13432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:44:38.553900 13432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:44:38.553993 13432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:44:38.554104 13432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:44:38.554207 13432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:44:38.554307 13432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:44:38.564157 13432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:46425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790332740062128-23952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.62:41681
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:41681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:44:38.565366 13432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:44:38.566625 13432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:44:38.578330 13437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:38.578389 13438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:38.578421 13440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:44:38.579123 13432 server_base.cc:1034] running on GCE node
I20260925 10:44:38.579905 13432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:44:38.581295 13432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:44:38.582516 13432 hybrid_clock.cc:648] HybridClock initialized: now 1790333078582474 us; error 50 us; skew 500 ppm
I20260925 10:44:38.582882 13432 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:44:38 dist-test-slave-dxdd krb5kdc[13419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333078, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:44:38.585807 13432 init.cc:377] Logged in from keytab as kudu/127.23.100.62@KRBTEST.COM (short username kudu)
I20260925 10:44:38.587182 13432 webserver.cc:460] Webserver started at http://127.23.100.62:44927/ using document root <none> and password file <none>
I20260925 10:44:38.587781 13432 fs_manager.cc:359] Metadata directory not provided
I20260925 10:44:38.587886 13432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:44:38.588122 13432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:44:38.590938 13432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790332740062128-23952-0/minicluster-data/master-0/data/instance:
uuid: "902e44c219954d6cb9e5135f79971f17"
format_stamp: "Formatted at 2026-09-25 10:44:38 on dist-test-slave-dxdd"
server_key: "548859350364076834cd3305b5e302e7"
server_key_iv: "5a542144267f29db6a843b2a7586bf51"
server_key_version: "encryptionkey@0"
I20260925 10:44:38.591634 13432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790332740062128-23952-0/minicluster-data/master-0/wal/instance:
uuid: "902e44c219954d6cb9e5135f79971f17"
format_stamp: "Formatted at 2026-09-25 10:44:38 on dist-test-slave-dxdd"
server_key: "548859350364076834cd3305b5e302e7"
server_key_iv: "5a542144267f29db6a843b2a7586bf51"
server_key_version: "encryptionkey@0"
I20260925 10:44:38.596452 13432 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 10:44:38.599670 13447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:44:38.600924 13432 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 10:44:38.601059 13432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790332740062128-23952-0/minicluster-data/master-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790332740062128-23952-0/minicluster-data/master-0/wal
uuid: "902e44c219954d6cb9e5135f79971f17"
format_stamp: "Formatted at 2026-09-25 10:44:38 on dist-test-slave-dxdd"
server_key: "548859350364076834cd3305b5e302e7"
server_key_iv: "5a542144267f29db6a843b2a7586bf51"
server_key_version: "encryptionkey@0"
I20260925 10:44:38.601217 13432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790332740062128-23952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790332740062128-23952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790332740062128-23952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:44:38.606472 13432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:44:38.610042 13432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:44:38.610275 13432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:44:38.628319 13432 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.62:41681
I20260925 10:44:38.628350 13498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.62:41681 every 8 connection(s)
I20260925 10:44:38.629644 13432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
I20260925 10:44:38.633277 13499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:44:38.635987 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 13432
I20260925 10:44:38.636152 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790332740062128-23952-0/minicluster-data/master-0/wal/instance
I20260925 10:44:38.636667 23952 external_mini_cluster.cc:1428] Setting key 7ea2731f294e2d421ee7192f9fc928cd
I20260925 10:44:38.644170 13502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:38.638466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34637 (local address 127.23.100.62:41681)
0925 10:44:38.639250 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:38.639262 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:38.639300 (+ 38us) server_negotiation.cc:407] Connection header received
0925 10:44:38.640880 (+ 1580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:38.640900 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:38.641220 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:38.641609 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:38.642651 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:38.642657 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:38.642672 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:44:38.642690 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:38.642749 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:44:38.642767 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:38.643032 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:38.643162 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 10:44:38.643340 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":332,"threads_started":1}
I20260925 10:44:38.644838 13499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 902e44c219954d6cb9e5135f79971f17: Bootstrap starting.
I20260925 10:44:38.647544 23952 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:44:38.647907 13499 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 902e44c219954d6cb9e5135f79971f17: Neither blocks nor log segments found. Creating new log.
I20260925 10:44:38.648876 13499 log.cc:824] T 00000000000000000000000000000000 P 902e44c219954d6cb9e5135f79971f17: Log is configured to *not* fsync() on all Append() calls
I20260925 10:44:38.651821 13499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 902e44c219954d6cb9e5135f79971f17: No bootstrap required, opened a new log
I20260925 10:44:38.654199 13502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:38.651671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60291 (local address 127.23.100.62:41681)
0925 10:44:38.651857 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:38.651866 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:38.652399 (+ 533us) server_negotiation.cc:407] Connection header received
0925 10:44:38.652565 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:38.652569 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:38.652608 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:38.652686 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:38.653575 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:38.653579 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:38.653582 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:44:38.653595 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:38.653632 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:44:38.653648 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:38.653746 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:38.653871 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 10:44:38.653918 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 10:44:38.659541 13499 raft_consensus.cc:348] T 00000000000000000000000000000000 P 902e44c219954d6cb9e5135f79971f17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902e44c219954d6cb9e5135f79971f17" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 41681 } }
I20260925 10:44:38.659914 13499 raft_consensus.cc:374] T 00000000000000000000000000000000 P 902e44c219954d6cb9e5135f79971f17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:44:38.660018 13499 raft_consensus.cc:729] T 00000000000000000000000000000000 P 902e44c219954d6cb9e5135f79971f17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 902e44c219954d6cb9e5135f79971f17, State: Initialized, Role: FOLLOWER
I20260925 10:44:38.660544 13499 consensus_queue.cc:260] T 00000000000000000000000000000000 P 902e44c219954d6cb9e5135f79971f17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902e44c219954d6cb9e5135f79971f17" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 41681 } }
I20260925 10:44:38.660772 13499 raft_consensus.cc:388] T 00000000000000000000000000000000 P 902e44c219954d6cb9e5135f79971f17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:44:38.660917 13499 raft_consensus.cc:482] T 00000000000000000000000000000000 P 902e44c219954d6cb9e5135f79971f17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:44:38.661094 13499 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 902e44c219954d6cb9e5135f79971f17 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:44:38.663242 13499 raft_consensus.cc:504] T 00000000000000000000000000000000 P 902e44c219954d6cb9e5135f79971f17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902e44c219954d6cb9e5135f79971f17" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 41681 } }
I20260925 10:44:38.663702 13499 leader_election.cc:302] T 00000000000000000000000000000000 P 902e44c219954d6cb9e5135f79971f17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 902e44c219954d6cb9e5135f79971f17; no voters:
I20260925 10:44:38.664232 13499 leader_election.cc:288] T 00000000000000000000000000000000 P 902e44c219954d6cb9e5135f79971f17 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:44:38.664361 13509 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 902e44c219954d6cb9e5135f79971f17 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:44:38.665051 13509 raft_consensus.cc:686] T 00000000000000000000000000000000 P 902e44c219954d6cb9e5135f79971f17 [term 1 LEADER]: Becoming Leader. State: Replica: 902e44c219954d6cb9e5135f79971f17, State: Running, Role: LEADER
I20260925 10:44:38.665557 13509 consensus_queue.cc:237] T 00000000000000000000000000000000 P 902e44c219954d6cb9e5135f79971f17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902e44c219954d6cb9e5135f79971f17" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 41681 } }
I20260925 10:44:38.666225 13499 sys_catalog.cc:564] T 00000000000000000000000000000000 P 902e44c219954d6cb9e5135f79971f17 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:44:38.668782 13510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 902e44c219954d6cb9e5135f79971f17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "902e44c219954d6cb9e5135f79971f17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902e44c219954d6cb9e5135f79971f17" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 41681 } } }
I20260925 10:44:38.669333 13510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 902e44c219954d6cb9e5135f79971f17 [sys.catalog]: This master's current role is: LEADER
I20260925 10:44:38.669250 13511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 902e44c219954d6cb9e5135f79971f17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 902e44c219954d6cb9e5135f79971f17. Latest consensus state: current_term: 1 leader_uuid: "902e44c219954d6cb9e5135f79971f17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902e44c219954d6cb9e5135f79971f17" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 41681 } } }
I20260925 10:44:38.669512 13511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 902e44c219954d6cb9e5135f79971f17 [sys.catalog]: This master's current role is: LEADER
I20260925 10:44:38.670284 13517 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:44:38.676275 13517 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:44:38.684146 13517 catalog_manager.cc:1347] Generated new cluster ID: c073a6d9006b432d8e7c8b082d1a9503
I20260925 10:44:38.684257 13517 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:44:38.715127 13517 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:44:38.715591 13517 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:44:38.720485 13517 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 902e44c219954d6cb9e5135f79971f17: Generated new TSK 0
I20260925 10:44:38.721179 13517 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:44:38.731987 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 13432
W20260925 10:44:38.732908 13494 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34637
W20260925 10:44:38.733536 23952 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 13432: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:44:38.733683 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 13432
2026-09-25T10:44:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1867 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790332740062128-23952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:44:38 dist-test-slave-dxdd krb5kdc[13530](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:44:38 dist-test-slave-dxdd krb5kdc[13530](info): set up 2 sockets
Sep 25 10:44:38 dist-test-slave-dxdd krb5kdc[13530](info): commencing operation
krb5kdc: starting...
W20260925 10:44:39.818797 23952 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.006s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Sep 25 10:44:39 dist-test-slave-dxdd krb5kdc[13530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:44:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:44:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
I20260925 10:44:40.057052 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:37239
--webserver_interface=127.23.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:38231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.62:37239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:44:40.374428 13543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:44:40.374704 13543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:44:40.374791 13543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:44:40.383176 13543 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:44:40.383279 13543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:44:40.383322 13543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:44:40.383381 13543 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:44:40.383422 13543 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:44:40.393055 13543 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:38231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790332740062128-23952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.62:37239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:37239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:44:40.394210 13543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:44:40.395520 13543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:44:40.406687 13548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:40.406719 13549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:40.406801 13551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:44:40.407397 13543 server_base.cc:1034] running on GCE node
I20260925 10:44:40.408022 13543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:44:40.409214 13543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:44:40.410396 13543 hybrid_clock.cc:648] HybridClock initialized: now 1790333080410356 us; error 40 us; skew 500 ppm
I20260925 10:44:40.410720 13543 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:44:40 dist-test-slave-dxdd krb5kdc[13530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333080, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:44:40.418990 13543 init.cc:377] Logged in from keytab as kudu/127.23.100.62@KRBTEST.COM (short username kudu)
I20260925 10:44:40.420394 13543 webserver.cc:460] Webserver started at http://127.23.100.62:45831/ using document root <none> and password file <none>
I20260925 10:44:40.421006 13543 fs_manager.cc:359] Metadata directory not provided
I20260925 10:44:40.421172 13543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:44:40.421459 13543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:44:40.424229 13543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790332740062128-23952-0/minicluster-data/master-0/data/instance:
uuid: "04e13f785d944dc0b21392dd526fa430"
format_stamp: "Formatted at 2026-09-25 10:44:40 on dist-test-slave-dxdd"
server_key: "57ea47cc916537ba001617d0f81066ee"
server_key_iv: "7e91a0b2e4b240db4b617753717f6900"
server_key_version: "encryptionkey@0"
I20260925 10:44:40.424963 13543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790332740062128-23952-0/minicluster-data/master-0/wal/instance:
uuid: "04e13f785d944dc0b21392dd526fa430"
format_stamp: "Formatted at 2026-09-25 10:44:40 on dist-test-slave-dxdd"
server_key: "57ea47cc916537ba001617d0f81066ee"
server_key_iv: "7e91a0b2e4b240db4b617753717f6900"
server_key_version: "encryptionkey@0"
I20260925 10:44:40.429617 13543 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.006s
I20260925 10:44:40.432727 13558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:44:40.433903 13543 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 10:44:40.434077 13543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790332740062128-23952-0/minicluster-data/master-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790332740062128-23952-0/minicluster-data/master-0/wal
uuid: "04e13f785d944dc0b21392dd526fa430"
format_stamp: "Formatted at 2026-09-25 10:44:40 on dist-test-slave-dxdd"
server_key: "57ea47cc916537ba001617d0f81066ee"
server_key_iv: "7e91a0b2e4b240db4b617753717f6900"
server_key_version: "encryptionkey@0"
I20260925 10:44:40.434262 13543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790332740062128-23952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790332740062128-23952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790332740062128-23952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:44:40.459034 13543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:44:40.462641 13543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:44:40.462921 13543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:44:40.480631 13543 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.62:37239
I20260925 10:44:40.480654 13609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.62:37239 every 8 connection(s)
I20260925 10:44:40.481858 13543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
I20260925 10:44:40.485296 13610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:44:40.490336 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 13543
I20260925 10:44:40.490499 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790332740062128-23952-0/minicluster-data/master-0/wal/instance
I20260925 10:44:40.491037 23952 external_mini_cluster.cc:1428] Setting key 7dc06de6bb4f1d902a3c3dfad23a4cc4
I20260925 10:44:40.495648 13610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04e13f785d944dc0b21392dd526fa430: Bootstrap starting.
I20260925 10:44:40.499174 13610 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04e13f785d944dc0b21392dd526fa430: Neither blocks nor log segments found. Creating new log.
I20260925 10:44:40.500368 13610 log.cc:824] T 00000000000000000000000000000000 P 04e13f785d944dc0b21392dd526fa430: Log is configured to *not* fsync() on all Append() calls
I20260925 10:44:40.503751 13610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04e13f785d944dc0b21392dd526fa430: No bootstrap required, opened a new log
Sep 25 10:44:40 dist-test-slave-dxdd krb5kdc[13530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:44:40.511612 13610 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04e13f785d944dc0b21392dd526fa430 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04e13f785d944dc0b21392dd526fa430" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 37239 } }
I20260925 10:44:40.511994 13610 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04e13f785d944dc0b21392dd526fa430 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:44:40.512112 13610 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04e13f785d944dc0b21392dd526fa430 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04e13f785d944dc0b21392dd526fa430, State: Initialized, Role: FOLLOWER
I20260925 10:44:40.512814 13610 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04e13f785d944dc0b21392dd526fa430 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04e13f785d944dc0b21392dd526fa430" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 37239 } }
I20260925 10:44:40.513098 13610 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04e13f785d944dc0b21392dd526fa430 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:44:40.513230 13610 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04e13f785d944dc0b21392dd526fa430 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:44:40.513413 13610 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04e13f785d944dc0b21392dd526fa430 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:44:40.514418 13613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:40.492680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49117 (local address 127.23.100.62:37239)
0925 10:44:40.493459 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:40.493469 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:40.493509 (+ 40us) server_negotiation.cc:407] Connection header received
0925 10:44:40.495135 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:40.495157 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:40.495499 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:40.495861 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:40.497534 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:40.498850 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:40.500625 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:40.501679 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:40.501828 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:40.506583 (+ 4755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:40.506590 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:40.506603 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:44:40.506641 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:40.509803 (+ 3162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:40.510624 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:40.510636 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:40.510642 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:40.510805 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:40.511210 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:40.511215 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:40.511220 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:40.511707 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:40.512025 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:40.512523 (+ 498us) server_negotiation.cc:299] Negotiation successful
0925 10:44:40.512980 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":363,"threads_started":1}
I20260925 10:44:40.516480 13610 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04e13f785d944dc0b21392dd526fa430 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04e13f785d944dc0b21392dd526fa430" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 37239 } }
I20260925 10:44:40.517010 13610 leader_election.cc:302] T 00000000000000000000000000000000 P 04e13f785d944dc0b21392dd526fa430 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04e13f785d944dc0b21392dd526fa430; no voters:
I20260925 10:44:40.517652 13610 leader_election.cc:288] T 00000000000000000000000000000000 P 04e13f785d944dc0b21392dd526fa430 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:44:40.517762 13615 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04e13f785d944dc0b21392dd526fa430 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:44:40.518445 23952 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:44:40.518452 13615 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04e13f785d944dc0b21392dd526fa430 [term 1 LEADER]: Becoming Leader. State: Replica: 04e13f785d944dc0b21392dd526fa430, State: Running, Role: LEADER
I20260925 10:44:40.519032 13615 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04e13f785d944dc0b21392dd526fa430 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04e13f785d944dc0b21392dd526fa430" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 37239 } }
I20260925 10:44:40.520689 13610 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04e13f785d944dc0b21392dd526fa430 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:44:40.521953 13617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04e13f785d944dc0b21392dd526fa430 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04e13f785d944dc0b21392dd526fa430" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04e13f785d944dc0b21392dd526fa430" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 37239 } } }
I20260925 10:44:40.521889 13616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04e13f785d944dc0b21392dd526fa430 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04e13f785d944dc0b21392dd526fa430. Latest consensus state: current_term: 1 leader_uuid: "04e13f785d944dc0b21392dd526fa430" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04e13f785d944dc0b21392dd526fa430" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 37239 } } }
I20260925 10:44:40.522428 13616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04e13f785d944dc0b21392dd526fa430 [sys.catalog]: This master's current role is: LEADER
I20260925 10:44:40.522428 13617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04e13f785d944dc0b21392dd526fa430 [sys.catalog]: This master's current role is: LEADER
I20260925 10:44:40.529268 13622 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:44:40.534658 13622 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:44:40.544735 13622 catalog_manager.cc:1347] Generated new cluster ID: 07d56590eb98486cb1347bfa4b84efc7
I20260925 10:44:40.544839 13622 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:44:40.547561 13613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:40.525773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44677 (local address 127.23.100.62:37239)
0925 10:44:40.527886 (+ 2113us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:40.527896 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:40.528864 (+ 968us) server_negotiation.cc:407] Connection header received
0925 10:44:40.529109 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:40.529116 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:40.529214 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:40.529324 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:40.531144 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:40.532302 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:40.534081 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:40.534885 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:40.535035 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:40.538102 (+ 3067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:40.538108 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:40.538113 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:44:40.538149 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:40.544585 (+ 6436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:40.545425 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:40.545433 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:40.545443 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:40.545606 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:40.546045 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:40.546050 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:40.546053 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:40.546445 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:40.546676 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:40.547034 (+ 358us) server_negotiation.cc:299] Negotiation successful
0925 10:44:40.547240 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1949}
I20260925 10:44:40.559130 13622 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:44:40.560462 13622 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:44:40.568243 13622 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04e13f785d944dc0b21392dd526fa430: Generated new TSK 0
I20260925 10:44:40.569031 13622 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:44:40.580657 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 13543
I20260925 10:44:40.582012 13605 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:44:40.669546 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 13543
2026-09-25T10:44:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1935 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790332740062128-23952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:44:40 dist-test-slave-dxdd krb5kdc[13642](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:44:40 dist-test-slave-dxdd krb5kdc[13642](info): set up 2 sockets
Sep 25 10:44:40 dist-test-slave-dxdd krb5kdc[13642](info): commencing operation
krb5kdc: starting...
W20260925 10:44:41.770953 23952 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Sep 25 10:44:41 dist-test-slave-dxdd krb5kdc[13642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:44:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:44:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
I20260925 10:44:42.019518 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:37641
--webserver_interface=127.23.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:38025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.62:37641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:44:42.337642 13655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:44:42.337921 13655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:44:42.338011 13655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:44:42.344980 13655 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:44:42.345134 13655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:44:42.345225 13655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:44:42.345329 13655 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:44:42.345400 13655 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:44:42.354687 13655 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:38025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790332740062128-23952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.62:37641
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:37641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:44:42.355854 13655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:44:42.357053 13655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:44:42.368229 13660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:42.368310 13661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:42.368273 13663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:44:42.369253 13655 server_base.cc:1034] running on GCE node
I20260925 10:44:42.369913 13655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:44:42.371187 13655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:44:42.372373 13655 hybrid_clock.cc:648] HybridClock initialized: now 1790333082372336 us; error 29 us; skew 500 ppm
I20260925 10:44:42.372690 13655 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:44:42 dist-test-slave-dxdd krb5kdc[13642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333082, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:44:42.375620 13655 init.cc:377] Logged in from keytab as kudu/127.23.100.62@KRBTEST.COM (short username kudu)
I20260925 10:44:42.376957 13655 webserver.cc:460] Webserver started at http://127.23.100.62:33791/ using document root <none> and password file <none>
I20260925 10:44:42.377583 13655 fs_manager.cc:359] Metadata directory not provided
I20260925 10:44:42.377740 13655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:44:42.378027 13655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:44:42.380862 13655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790332740062128-23952-0/minicluster-data/master-0/data/instance:
uuid: "48264c0498be41328f4c93355dce9b81"
format_stamp: "Formatted at 2026-09-25 10:44:42 on dist-test-slave-dxdd"
server_key: "c34f2b93eb0fc73af0e627e67af8f09a"
server_key_iv: "8ac260fb7edec2ab126ae001edd7c384"
server_key_version: "encryptionkey@0"
I20260925 10:44:42.381582 13655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790332740062128-23952-0/minicluster-data/master-0/wal/instance:
uuid: "48264c0498be41328f4c93355dce9b81"
format_stamp: "Formatted at 2026-09-25 10:44:42 on dist-test-slave-dxdd"
server_key: "c34f2b93eb0fc73af0e627e67af8f09a"
server_key_iv: "8ac260fb7edec2ab126ae001edd7c384"
server_key_version: "encryptionkey@0"
I20260925 10:44:42.387571 13655 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260925 10:44:42.390839 13670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:44:42.392107 13655 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:44:42.392262 13655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790332740062128-23952-0/minicluster-data/master-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790332740062128-23952-0/minicluster-data/master-0/wal
uuid: "48264c0498be41328f4c93355dce9b81"
format_stamp: "Formatted at 2026-09-25 10:44:42 on dist-test-slave-dxdd"
server_key: "c34f2b93eb0fc73af0e627e67af8f09a"
server_key_iv: "8ac260fb7edec2ab126ae001edd7c384"
server_key_version: "encryptionkey@0"
I20260925 10:44:42.392433 13655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790332740062128-23952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790332740062128-23952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790332740062128-23952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:44:42.417430 13655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:44:42.421175 13655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:44:42.421464 13655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:44:42.439286 13655 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.62:37641
I20260925 10:44:42.439306 13721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.62:37641 every 8 connection(s)
I20260925 10:44:42.440574 13655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
I20260925 10:44:42.443207 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 13655
I20260925 10:44:42.443395 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790332740062128-23952-0/minicluster-data/master-0/wal/instance
I20260925 10:44:42.443810 23952 external_mini_cluster.cc:1428] Setting key e96501b9c125ed10dacc0dcc50d2dab0
I20260925 10:44:42.444207 13722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:44:42.457064 13725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:42.445449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52973 (local address 127.23.100.62:37641)
0925 10:44:42.446232 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:42.446242 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:42.446279 (+ 37us) server_negotiation.cc:407] Connection header received
0925 10:44:42.448644 (+ 2365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:42.448664 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:42.449013 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:42.449304 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:42.449847 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:42.451223 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:42.452977 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:42.453644 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:42.453779 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:42.455002 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:42.455012 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:42.455024 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:44:42.455540 (+ 516us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:42.455615 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:44:42.455637 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:42.455721 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:42.455932 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 10:44:42.456168 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":598,"thread_start_us":379,"threads_started":1}
I20260925 10:44:42.457283 13722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48264c0498be41328f4c93355dce9b81: Bootstrap starting.
I20260925 10:44:42.460326 13722 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48264c0498be41328f4c93355dce9b81: Neither blocks nor log segments found. Creating new log.
I20260925 10:44:42.460466 23952 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:44:42.461267 13722 log.cc:824] T 00000000000000000000000000000000 P 48264c0498be41328f4c93355dce9b81: Log is configured to *not* fsync() on all Append() calls
I20260925 10:44:42.463814 13722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48264c0498be41328f4c93355dce9b81: No bootstrap required, opened a new log
I20260925 10:44:42.471148 13725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:42.464790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49549 (local address 127.23.100.62:37641)
0925 10:44:42.464989 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:42.464994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:42.465337 (+ 343us) server_negotiation.cc:407] Connection header received
0925 10:44:42.465506 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:42.465509 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:42.465563 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:42.465658 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:42.466210 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:42.466924 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:42.468736 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:42.469194 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:42.469283 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:42.470420 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:42.470424 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:42.470428 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:44:42.470446 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:42.470480 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:44:42.470494 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:42.470558 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:42.470740 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 10:44:42.470806 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 10:44:42.473111 13722 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48264c0498be41328f4c93355dce9b81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48264c0498be41328f4c93355dce9b81" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 37641 } }
I20260925 10:44:42.473448 13722 raft_consensus.cc:374] T 00000000000000000000000000000000 P 48264c0498be41328f4c93355dce9b81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:44:42.473553 13722 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48264c0498be41328f4c93355dce9b81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48264c0498be41328f4c93355dce9b81, State: Initialized, Role: FOLLOWER
I20260925 10:44:42.474042 13722 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48264c0498be41328f4c93355dce9b81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48264c0498be41328f4c93355dce9b81" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 37641 } }
I20260925 10:44:42.474270 13722 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48264c0498be41328f4c93355dce9b81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:44:42.474429 13722 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48264c0498be41328f4c93355dce9b81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:44:42.474594 13722 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48264c0498be41328f4c93355dce9b81 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:44:42.476670 13722 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48264c0498be41328f4c93355dce9b81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48264c0498be41328f4c93355dce9b81" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 37641 } }
I20260925 10:44:42.477120 13722 leader_election.cc:302] T 00000000000000000000000000000000 P 48264c0498be41328f4c93355dce9b81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48264c0498be41328f4c93355dce9b81; no voters:
I20260925 10:44:42.477663 13722 leader_election.cc:288] T 00000000000000000000000000000000 P 48264c0498be41328f4c93355dce9b81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:44:42.477802 13732 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48264c0498be41328f4c93355dce9b81 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:44:42.478514 13732 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48264c0498be41328f4c93355dce9b81 [term 1 LEADER]: Becoming Leader. State: Replica: 48264c0498be41328f4c93355dce9b81, State: Running, Role: LEADER
I20260925 10:44:42.479197 13732 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48264c0498be41328f4c93355dce9b81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48264c0498be41328f4c93355dce9b81" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 37641 } }
I20260925 10:44:42.479898 13722 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48264c0498be41328f4c93355dce9b81 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:44:42.482034 13733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48264c0498be41328f4c93355dce9b81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48264c0498be41328f4c93355dce9b81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48264c0498be41328f4c93355dce9b81" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 37641 } } }
I20260925 10:44:42.482431 13733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48264c0498be41328f4c93355dce9b81 [sys.catalog]: This master's current role is: LEADER
I20260925 10:44:42.482133 13734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48264c0498be41328f4c93355dce9b81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48264c0498be41328f4c93355dce9b81. Latest consensus state: current_term: 1 leader_uuid: "48264c0498be41328f4c93355dce9b81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48264c0498be41328f4c93355dce9b81" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 37641 } } }
I20260925 10:44:42.482777 13734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48264c0498be41328f4c93355dce9b81 [sys.catalog]: This master's current role is: LEADER
I20260925 10:44:42.487237 13738 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:44:42.491686 13738 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:44:42.500891 13738 catalog_manager.cc:1347] Generated new cluster ID: beb058a49f7c46eb8a023d6b7fc38645
I20260925 10:44:42.500999 13738 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:44:42.520944 13738 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:44:42.522140 13738 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:44:42.532742 13738 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48264c0498be41328f4c93355dce9b81: Generated new TSK 0
I20260925 10:44:42.533455 13738 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:44:42.546578 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 13655
W20260925 10:44:42.547524 13717 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52973
W20260925 10:44:42.548130 23952 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 13655: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:44:42.548259 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 13655
2026-09-25T10:44:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1878 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790332740062128-23952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:44:42 dist-test-slave-dxdd krb5kdc[13753](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:44:42 dist-test-slave-dxdd krb5kdc[13753](info): set up 2 sockets
Sep 25 10:44:42 dist-test-slave-dxdd krb5kdc[13753](info): commencing operation
krb5kdc: starting...
W20260925 10:44:43.634824 23952 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Sep 25 10:44:43 dist-test-slave-dxdd krb5kdc[13753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:44:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:44:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
I20260925 10:44:43.868975 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:33191
--webserver_interface=127.23.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:40075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.62:33191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:44:44.183982 13766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:44:44.184242 13766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:44:44.184331 13766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:44:44.191267 13766 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:44:44.191385 13766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:44:44.191442 13766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:44:44.191517 13766 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:44:44.191560 13766 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:44:44.200901 13766 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:40075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790332740062128-23952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.62:33191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:33191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:44:44.202034 13766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:44:44.203276 13766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:44:44.218389 13772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:44.218390 13774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:44.218700 13771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:44:44.219434 13766 server_base.cc:1034] running on GCE node
I20260925 10:44:44.220221 13766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:44:44.221539 13766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:44:44.222728 13766 hybrid_clock.cc:648] HybridClock initialized: now 1790333084222675 us; error 47 us; skew 500 ppm
I20260925 10:44:44.223039 13766 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:44:44 dist-test-slave-dxdd krb5kdc[13753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333084, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:44:44.225924 13766 init.cc:377] Logged in from keytab as kudu/127.23.100.62@KRBTEST.COM (short username kudu)
I20260925 10:44:44.227253 13766 webserver.cc:460] Webserver started at http://127.23.100.62:46633/ using document root <none> and password file <none>
I20260925 10:44:44.227841 13766 fs_manager.cc:359] Metadata directory not provided
I20260925 10:44:44.227949 13766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:44:44.228191 13766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:44:44.231179 13766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790332740062128-23952-0/minicluster-data/master-0/data/instance:
uuid: "68944dfc5fff4479a7f5127d4023d240"
format_stamp: "Formatted at 2026-09-25 10:44:44 on dist-test-slave-dxdd"
server_key: "ce84eba4327a6d724ec2be4e6bb1a2e3"
server_key_iv: "49f331046b7b885c5770b890c56fb02f"
server_key_version: "encryptionkey@0"
I20260925 10:44:44.231863 13766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790332740062128-23952-0/minicluster-data/master-0/wal/instance:
uuid: "68944dfc5fff4479a7f5127d4023d240"
format_stamp: "Formatted at 2026-09-25 10:44:44 on dist-test-slave-dxdd"
server_key: "ce84eba4327a6d724ec2be4e6bb1a2e3"
server_key_iv: "49f331046b7b885c5770b890c56fb02f"
server_key_version: "encryptionkey@0"
I20260925 10:44:44.236696 13766 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 10:44:44.239965 13781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:44:44.241279 13766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 10:44:44.241463 13766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790332740062128-23952-0/minicluster-data/master-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790332740062128-23952-0/minicluster-data/master-0/wal
uuid: "68944dfc5fff4479a7f5127d4023d240"
format_stamp: "Formatted at 2026-09-25 10:44:44 on dist-test-slave-dxdd"
server_key: "ce84eba4327a6d724ec2be4e6bb1a2e3"
server_key_iv: "49f331046b7b885c5770b890c56fb02f"
server_key_version: "encryptionkey@0"
I20260925 10:44:44.241652 13766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790332740062128-23952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790332740062128-23952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790332740062128-23952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:44:44.247314 13766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:44:44.251173 13766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:44:44.251468 13766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:44:44.269601 13766 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.62:33191
I20260925 10:44:44.269631 13832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.62:33191 every 8 connection(s)
I20260925 10:44:44.271106 13766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
I20260925 10:44:44.274787 13833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:44:44.280931 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 13766
I20260925 10:44:44.281111 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790332740062128-23952-0/minicluster-data/master-0/wal/instance
I20260925 10:44:44.281570 23952 external_mini_cluster.cc:1428] Setting key e4aec18e1850475864e89464419b88c9
I20260925 10:44:44.285641 13833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68944dfc5fff4479a7f5127d4023d240: Bootstrap starting.
I20260925 10:44:44.289633 13833 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68944dfc5fff4479a7f5127d4023d240: Neither blocks nor log segments found. Creating new log.
I20260925 10:44:44.290551 13836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:44.283497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33183 (local address 127.23.100.62:33191)
0925 10:44:44.284310 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:44.284324 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:44.284369 (+ 45us) server_negotiation.cc:407] Connection header received
0925 10:44:44.286797 (+ 2428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:44.286824 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:44.287272 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:44.287745 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:44.288702 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:44.288711 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:44.288727 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:44:44.288746 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:44.288818 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:44:44.288850 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:44.289033 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:44.289148 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 10:44:44.289322 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":312,"threads_started":1}
I20260925 10:44:44.290951 13833 log.cc:824] T 00000000000000000000000000000000 P 68944dfc5fff4479a7f5127d4023d240: Log is configured to *not* fsync() on all Append() calls
I20260925 10:44:44.294091 13833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68944dfc5fff4479a7f5127d4023d240: No bootstrap required, opened a new log
I20260925 10:44:44.294435 23952 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:44:44.300992 13836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:44.298623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49047 (local address 127.23.100.62:33191)
0925 10:44:44.298808 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:44.298816 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:44.299233 (+ 417us) server_negotiation.cc:407] Connection header received
0925 10:44:44.299352 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:44.299358 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:44.299420 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:44.299544 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:44.300378 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:44.300383 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:44.300388 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:44:44.300409 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:44.300456 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:44:44.300482 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:44.300610 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:44.300655 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 10:44:44.300704 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 10:44:44.302141 13833 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68944dfc5fff4479a7f5127d4023d240 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68944dfc5fff4479a7f5127d4023d240" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33191 } }
I20260925 10:44:44.302569 13833 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68944dfc5fff4479a7f5127d4023d240 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:44:44.302672 13833 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68944dfc5fff4479a7f5127d4023d240 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68944dfc5fff4479a7f5127d4023d240, State: Initialized, Role: FOLLOWER
I20260925 10:44:44.303194 13833 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68944dfc5fff4479a7f5127d4023d240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68944dfc5fff4479a7f5127d4023d240" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33191 } }
I20260925 10:44:44.303431 13833 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68944dfc5fff4479a7f5127d4023d240 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:44:44.303582 13833 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68944dfc5fff4479a7f5127d4023d240 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:44:44.303771 13833 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68944dfc5fff4479a7f5127d4023d240 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:44:44.306049 13833 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68944dfc5fff4479a7f5127d4023d240 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68944dfc5fff4479a7f5127d4023d240" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33191 } }
I20260925 10:44:44.306545 13833 leader_election.cc:302] T 00000000000000000000000000000000 P 68944dfc5fff4479a7f5127d4023d240 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68944dfc5fff4479a7f5127d4023d240; no voters:
I20260925 10:44:44.307126 13833 leader_election.cc:288] T 00000000000000000000000000000000 P 68944dfc5fff4479a7f5127d4023d240 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:44:44.307250 13843 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68944dfc5fff4479a7f5127d4023d240 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:44:44.308872 13843 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68944dfc5fff4479a7f5127d4023d240 [term 1 LEADER]: Becoming Leader. State: Replica: 68944dfc5fff4479a7f5127d4023d240, State: Running, Role: LEADER
I20260925 10:44:44.309366 13843 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68944dfc5fff4479a7f5127d4023d240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68944dfc5fff4479a7f5127d4023d240" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33191 } }
I20260925 10:44:44.309486 13833 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68944dfc5fff4479a7f5127d4023d240 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:44:44.312054 13844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68944dfc5fff4479a7f5127d4023d240 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68944dfc5fff4479a7f5127d4023d240" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68944dfc5fff4479a7f5127d4023d240" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33191 } } }
I20260925 10:44:44.312027 13845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68944dfc5fff4479a7f5127d4023d240 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68944dfc5fff4479a7f5127d4023d240. Latest consensus state: current_term: 1 leader_uuid: "68944dfc5fff4479a7f5127d4023d240" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68944dfc5fff4479a7f5127d4023d240" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 33191 } } }
I20260925 10:44:44.312500 13845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68944dfc5fff4479a7f5127d4023d240 [sys.catalog]: This master's current role is: LEADER
I20260925 10:44:44.312500 13844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68944dfc5fff4479a7f5127d4023d240 [sys.catalog]: This master's current role is: LEADER
I20260925 10:44:44.313524 13850 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:44:44.319552 13850 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:44:44.328181 13850 catalog_manager.cc:1347] Generated new cluster ID: 7e7e108e0c5343da9cf57f3a7da2962f
I20260925 10:44:44.328313 13850 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:44:44.351913 13850 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:44:44.352383 13850 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:44:44.362543 13850 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68944dfc5fff4479a7f5127d4023d240: Generated new TSK 0
I20260925 10:44:44.363306 13850 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:44:44.369105 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 13766
W20260925 10:44:44.370087 13828 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33183
W20260925 10:44:44.370810 23952 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 13766: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:44:44.370952 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 13766
2026-09-25T10:44:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1822 ms)
[----------] 18 tests from AuthTokenIssuingTest (33922 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:44:44 dist-test-slave-dxdd krb5kdc[13864](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:44:44 dist-test-slave-dxdd krb5kdc[13864](info): set up 2 sockets
Sep 25 10:44:44 dist-test-slave-dxdd krb5kdc[13864](info): commencing operation
krb5kdc: starting...
W20260925 10:44:45.458843 23952 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Sep 25 10:44:45 dist-test-slave-dxdd krb5kdc[13864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:44:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:44:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
I20260925 10:44:45.703712 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:40919
--webserver_interface=127.23.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:38949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.62:40919,127.23.100.61:33291,127.23.100.60:35033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:44:46.019748 13877 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 10:44:46.020076 13877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:44:46.020161 13877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:44:46.020246 13877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:44:46.027406 13877 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:44:46.027499 13877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:44:46.027557 13877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:44:46.027608 13877 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:44:46.027663 13877 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:44:46.037218 13877 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:38949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.62:40919,127.23.100.61:33291,127.23.100.60:35033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:40919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:44:46.038455 13877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:44:46.039705 13877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:44:46.050999 13882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:46.051898 13883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:46.052341 13885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:44:46.052699 13877 server_base.cc:1034] running on GCE node
I20260925 10:44:46.053937 13877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:44:46.056331 13877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:44:46.057854 13877 hybrid_clock.cc:648] HybridClock initialized: now 1790333086057694 us; error 117 us; skew 500 ppm
I20260925 10:44:46.058266 13877 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:44:46 dist-test-slave-dxdd krb5kdc[13864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333086, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:44:46.063391 13877 init.cc:377] Logged in from keytab as kudu/127.23.100.62@KRBTEST.COM (short username kudu)
I20260925 10:44:46.065156 13877 webserver.cc:460] Webserver started at http://127.23.100.62:38587/ using document root <none> and password file <none>
I20260925 10:44:46.065999 13877 fs_manager.cc:359] Metadata directory not provided
I20260925 10:44:46.066160 13877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:44:46.066587 13877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:44:46.070130 13877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/minicluster-data/master-0/data/instance:
uuid: "73d518b5b1974a44add4bea797e5d9f5"
format_stamp: "Formatted at 2026-09-25 10:44:46 on dist-test-slave-dxdd"
server_key: "3fd024cdcad0ad765f008c0ef66e353d"
server_key_iv: "1fd45253a97cd22f59699868d6f91d7f"
server_key_version: "encryptionkey@0"
I20260925 10:44:46.071192 13877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/minicluster-data/master-0/wal/instance:
uuid: "73d518b5b1974a44add4bea797e5d9f5"
format_stamp: "Formatted at 2026-09-25 10:44:46 on dist-test-slave-dxdd"
server_key: "3fd024cdcad0ad765f008c0ef66e353d"
server_key_iv: "1fd45253a97cd22f59699868d6f91d7f"
server_key_version: "encryptionkey@0"
I20260925 10:44:46.078352 13877 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260925 10:44:46.083019 13892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:44:46.084715 13877 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260925 10:44:46.084899 13877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/minicluster-data/master-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/minicluster-data/master-0/wal
uuid: "73d518b5b1974a44add4bea797e5d9f5"
format_stamp: "Formatted at 2026-09-25 10:44:46 on dist-test-slave-dxdd"
server_key: "3fd024cdcad0ad765f008c0ef66e353d"
server_key_iv: "1fd45253a97cd22f59699868d6f91d7f"
server_key_version: "encryptionkey@0"
I20260925 10:44:46.085135 13877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:44:46.107659 13877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:44:46.112370 13877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:44:46.112653 13877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:44:46.130859 13877 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.62:40919
I20260925 10:44:46.130900 13943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.62:40919 every 8 connection(s)
I20260925 10:44:46.132164 13877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
I20260925 10:44:46.135699 13944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:44:46.136363 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 13877
I20260925 10:44:46.136602 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/minicluster-data/master-0/wal/instance
I20260925 10:44:46.137229 23952 external_mini_cluster.cc:1428] Setting key 15fa0ee7e0fa875c752aa624dc441f17
I20260925 10:44:46.142100 13944 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.62" port: 40919 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:44:46 dist-test-slave-dxdd krb5kdc[13864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333086, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.62@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.100.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.61@KRBTEST.COM" created.
I20260925 10:44:46.166230 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:46.145173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:40919 (local address 127.0.0.1:60898)
0925 10:44:46.145960 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:46.146005 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 10:44:46.148009 (+ 2004us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:44:46.148916 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:44:46.148921 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:44:46.149427 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:44:46.149766 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:46.149782 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:46.152067 (+ 2285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:46.152071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:46.153090 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:46.153100 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:46.154630 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:46.154636 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:46.154766 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:46.155417 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:44:46.155438 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:44:46.157733 (+ 2295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:44:46.162104 (+ 4371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:44:46.162113 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:44:46.162121 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:44:46.162635 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:44:46.163077 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:44:46.163080 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:44:46.163082 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:44:46.163223 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:44:46.163977 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:44:46.163982 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:44:46.164153 (+ 171us) client_negotiation.cc:769] Sending connection context
0925 10:44:46.164348 (+ 195us) client_negotiation.cc:240] Negotiation successful
0925 10:44:46.164813 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":356,"threads_started":1}
I20260925 10:44:46.166265 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:46.146435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60898 (local address 127.23.100.62:40919)
0925 10:44:46.147112 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:46.147123 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:46.147174 (+ 51us) server_negotiation.cc:407] Connection header received
0925 10:44:46.148368 (+ 1194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:46.148376 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:46.148679 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:46.148902 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:46.149982 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:46.151908 (+ 1926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:46.153227 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:46.154179 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:46.154404 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:46.157977 (+ 3573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:46.157987 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:46.158001 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:44:46.158041 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:46.161962 (+ 3921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:46.162768 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:46.162780 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:46.162785 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:46.162975 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:46.163321 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:46.163324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:46.163326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:46.163842 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:46.164007 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:46.164556 (+ 549us) server_negotiation.cc:299] Negotiation successful
0925 10:44:46.164943 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":127,"server-negotiator.queue_time_us":451,"thread_start_us":302,"threads_started":1}
I20260925 10:44:46.169247 13944 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.61" port: 33291 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:44:46.170353 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:46.169947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.61:33291 (local address 127.0.0.1:38382)
0925 10:44:46.170106 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:46.170212 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.61:33291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260925 10:44:46.170635 13893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.100.61:33291: connect: Connection refused (error 111)
I20260925 10:44:46.172101 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:46.171763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.61:33291 (local address 127.0.0.1:38392)
0925 10:44:46.171912 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:46.171975 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.61:33291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 10:44:46.172295 13944 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.61:33291: Network error: Client connection negotiation failed: client connection to 127.23.100.61:33291: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.61.keytab.
Entry for principal kudu/127.23.100.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.61.keytab.
I20260925 10:44:46.194900 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.61:33291
--webserver_interface=127.23.100.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.61.keytab
--principal=kudu/127.23.100.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:38949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.62:40919,127.23.100.61:33291,127.23.100.60:35033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:44:46.221626 13944 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.61" port: 33291 } attempt: 1
I20260925 10:44:46.222918 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:46.222357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.61:33291 (local address 127.0.0.1:38394)
0925 10:44:46.222611 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:46.222734 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.61:33291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260925 10:44:46.223872 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:46.223512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.61:33291 (local address 127.0.0.1:38406)
0925 10:44:46.223642 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:46.223721 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.61:33291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 10:44:46.224045 13944 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.61:33291: Network error: Client connection negotiation failed: client connection to 127.23.100.61:33291: connect: Connection refused (error 111)
I20260925 10:44:46.292297 13944 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.61" port: 33291 } attempt: 2
I20260925 10:44:46.293498 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:46.292977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.61:33291 (local address 127.0.0.1:38422)
0925 10:44:46.293227 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:46.293332 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.61:33291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260925 10:44:46.294525 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:46.294109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.61:33291 (local address 127.0.0.1:38434)
0925 10:44:46.294247 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:46.294374 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.61:33291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 10:44:46.294728 13944 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.61:33291: Network error: Client connection negotiation failed: client connection to 127.23.100.61:33291: connect: Connection refused (error 111)
I20260925 10:44:46.386008 13944 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.61" port: 33291 } attempt: 3
I20260925 10:44:46.387248 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:46.386745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.61:33291 (local address 127.0.0.1:38440)
0925 10:44:46.386976 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:46.387088 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.61:33291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260925 10:44:46.388355 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:46.387922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.61:33291 (local address 127.0.0.1:38448)
0925 10:44:46.388079 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:46.388160 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.61:33291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260925 10:44:46.388536 13944 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.61:33291: Network error: Client connection negotiation failed: client connection to 127.23.100.61:33291: connect: Connection refused (error 111)
W20260925 10:44:46.512467 13951 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 10:44:46.512826 13951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:44:46.512948 13951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:44:46.513134 13951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:44:46.520900 13951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:44:46.520982 13951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:44:46.521098 13951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:44:46.521183 13951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:44:46.521273 13951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:44:46.530722 13951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:38949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.62:40919,127.23.100.61:33291,127.23.100.60:35033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.61.keytab
--principal=kudu/127.23.100.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.61:33291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.100.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:44:46.531777 13944 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.61" port: 33291 } attempt: 4
I20260925 10:44:46.532058 13951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:44:46.533020 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:46.532419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.61:33291 (local address 127.0.0.1:38452)
0925 10:44:46.532726 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:46.532829 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.61:33291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
I20260925 10:44:46.533713 13951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 10:44:46.534041 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:46.533630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.61:33291 (local address 127.0.0.1:38464)
0925 10:44:46.533786 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:46.533891 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.61:33291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 10:44:46.534246 13944 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.61:33291: Network error: Client connection negotiation failed: client connection to 127.23.100.61:33291: connect: Connection refused (error 111)
W20260925 10:44:46.544916 13957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:46.544948 13956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:46.545363 13959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:44:46.546003 13951 server_base.cc:1034] running on GCE node
I20260925 10:44:46.546777 13951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:44:46.547986 13951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:44:46.549188 13951 hybrid_clock.cc:648] HybridClock initialized: now 1790333086549136 us; error 33 us; skew 500 ppm
I20260925 10:44:46.549494 13951 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:44:46 dist-test-slave-dxdd krb5kdc[13864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333086, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:44:46.552449 13951 init.cc:377] Logged in from keytab as kudu/127.23.100.61@KRBTEST.COM (short username kudu)
I20260925 10:44:46.553758 13951 webserver.cc:460] Webserver started at http://127.23.100.61:34477/ using document root <none> and password file <none>
I20260925 10:44:46.554448 13951 fs_manager.cc:359] Metadata directory not provided
I20260925 10:44:46.554571 13951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:44:46.554873 13951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:44:46.557602 13951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/minicluster-data/master-1/data/instance:
uuid: "b7672de60ddc48a087db664d20970e22"
format_stamp: "Formatted at 2026-09-25 10:44:46 on dist-test-slave-dxdd"
server_key: "ce60eca9f38aca017a8707d9017e8d6d"
server_key_iv: "2523782257595d9784dbd360b7bc441a"
server_key_version: "encryptionkey@0"
I20260925 10:44:46.558403 13951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/minicluster-data/master-1/wal/instance:
uuid: "b7672de60ddc48a087db664d20970e22"
format_stamp: "Formatted at 2026-09-25 10:44:46 on dist-test-slave-dxdd"
server_key: "ce60eca9f38aca017a8707d9017e8d6d"
server_key_iv: "2523782257595d9784dbd360b7bc441a"
server_key_version: "encryptionkey@0"
I20260925 10:44:46.563549 13951 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 10:44:46.566723 13966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:44:46.567914 13951 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 10:44:46.568091 13951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/minicluster-data/master-1/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/minicluster-data/master-1/wal
uuid: "b7672de60ddc48a087db664d20970e22"
format_stamp: "Formatted at 2026-09-25 10:44:46 on dist-test-slave-dxdd"
server_key: "ce60eca9f38aca017a8707d9017e8d6d"
server_key_iv: "2523782257595d9784dbd360b7bc441a"
server_key_version: "encryptionkey@0"
I20260925 10:44:46.568286 13951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:44:46.580791 13951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:44:46.584754 13951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:44:46.584950 13951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:44:46.601864 13951 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.61:33291
I20260925 10:44:46.601890 14017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.61:33291 every 8 connection(s)
I20260925 10:44:46.603188 13951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/minicluster-data/master-1/data/info.pb
I20260925 10:44:46.606724 14018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:44:46.607808 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 13951
I20260925 10:44:46.607985 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/minicluster-data/master-1/wal/instance
I20260925 10:44:46.608407 23952 external_mini_cluster.cc:1428] Setting key e44ac683d9a0e02b50ad2df32b54a747
I20260925 10:44:46.613083 14018 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.62" port: 40919 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:44:46 dist-test-slave-dxdd krb5kdc[13864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333086, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.61@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.100.60@KRBTEST.COM; defaulting to no policy
I20260925 10:44:46.633160 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:46.615724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60900 (local address 127.23.100.62:40919)
0925 10:44:46.615942 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:46.615949 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:46.617045 (+ 1096us) server_negotiation.cc:407] Connection header received
0925 10:44:46.618886 (+ 1841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:46.618895 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:46.618988 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:46.619109 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:46.620431 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:46.621531 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:46.623048 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:46.623841 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:46.623999 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:46.627198 (+ 3199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:46.627205 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:46.627229 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:44:46.627272 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:46.630097 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:46.631062 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:46.631067 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:46.631069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:46.631181 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:46.631608 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:46.631610 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:46.631612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:46.631948 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:46.632058 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:46.632723 (+ 665us) server_negotiation.cc:299] Negotiation successful
0925 10:44:46.632939 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 10:44:46.633893 14021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:46.616152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:40919 (local address 127.0.0.1:60900)
0925 10:44:46.616877 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:46.616920 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 10:44:46.618601 (+ 1681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:44:46.619277 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:44:46.619284 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:44:46.619761 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:44:46.620251 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:46.620266 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:46.621688 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:46.621692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:46.622886 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:46.622897 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:46.624043 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:46.624047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:46.624250 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:46.624906 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:44:46.624928 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:44:46.626978 (+ 2050us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:44:46.630237 (+ 3259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:44:46.630245 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:44:46.630261 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:44:46.630921 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:44:46.631289 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:44:46.631297 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:44:46.631302 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:44:46.631501 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:44:46.632069 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:44:46.632075 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:44:46.632358 (+ 283us) client_negotiation.cc:769] Sending connection context
0925 10:44:46.632565 (+ 207us) client_negotiation.cc:240] Negotiation successful
0925 10:44:46.632944 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":333,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.60@KRBTEST.COM" created.
I20260925 10:44:46.635607 14018 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.61" port: 33291 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:44:46 dist-test-slave-dxdd krb5kdc[13864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333086, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.61@KRBTEST.COM for kudu/127.23.100.61@KRBTEST.COM
I20260925 10:44:46.649701 14021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:46.636317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.61:33291 (local address 127.0.0.1:38476)
0925 10:44:46.636462 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:46.636476 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:44:46.636547 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:44:46.637535 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:44:46.637539 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:44:46.637921 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:44:46.638035 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:46.638042 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:46.639428 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:46.639436 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:46.640233 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:46.640243 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:46.641004 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:46.641009 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:46.641203 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:46.641737 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:44:46.641753 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:44:46.643817 (+ 2064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:44:46.647448 (+ 3631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:44:46.647453 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:44:46.647457 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:44:46.647775 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:44:46.648125 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:44:46.648132 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:44:46.648137 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:44:46.648267 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:44:46.648798 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:44:46.648802 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:44:46.648941 (+ 139us) client_negotiation.cc:769] Sending connection context
0925 10:44:46.649073 (+ 132us) client_negotiation.cc:240] Negotiation successful
0925 10:44:46.649329 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 10:44:46.649827 14022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:46.636572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38476 (local address 127.23.100.61:33291)
0925 10:44:46.637183 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:46.637188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:46.637210 (+ 22us) server_negotiation.cc:407] Connection header received
0925 10:44:46.637250 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:46.637255 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:46.637402 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:46.637551 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:46.638200 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:46.639296 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:46.640364 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:46.640860 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:46.641000 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:46.644054 (+ 3054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:46.644068 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:46.644086 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:44:46.644127 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:46.647291 (+ 3164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:46.647907 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:46.647914 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:46.647918 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:46.648025 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:46.648382 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:46.648386 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:46.648389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:46.648681 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:46.648880 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:46.649207 (+ 327us) server_negotiation.cc:299] Negotiation successful
0925 10:44:46.649448 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":491,"thread_start_us":379,"threads_started":1}
I20260925 10:44:46.652251 14018 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.60" port: 35033 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:44:46.653553 14021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:46.652975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:35033 (local address 127.0.0.1:46896)
0925 10:44:46.653196 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:46.653348 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.60:35033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260925 10:44:46.653856 13970 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.100.60:35033: connect: Connection refused (error 111)
I20260925 10:44:46.655445 14021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:46.655040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:35033 (local address 127.0.0.1:46908)
0925 10:44:46.655214 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:46.655314 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.60:35033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260925 10:44:46.655689 14018 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.60:35033: Network error: Client connection negotiation failed: client connection to 127.23.100.60:35033: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.60.keytab.
Entry for principal kudu/127.23.100.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.60.keytab.
I20260925 10:44:46.663756 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.60:35033
--webserver_interface=127.23.100.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.60.keytab
--principal=kudu/127.23.100.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:38949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.62:40919,127.23.100.61:33291,127.23.100.60:35033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:44:46.704985 14018 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.60" port: 35033 } attempt: 1
I20260925 10:44:46.706173 14021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:46.705611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:35033 (local address 127.0.0.1:46918)
0925 10:44:46.705831 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:46.705967 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.60:35033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 10:44:46.707297 14021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:46.706859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:35033 (local address 127.0.0.1:46922)
0925 10:44:46.707059 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:46.707150 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.60:35033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260925 10:44:46.707511 14018 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.60:35033: Network error: Client connection negotiation failed: client connection to 127.23.100.60:35033: connect: Connection refused (error 111)
I20260925 10:44:46.775748 14018 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.60" port: 35033 } attempt: 2
I20260925 10:44:46.776837 14021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:46.776349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:35033 (local address 127.0.0.1:46930)
0925 10:44:46.776553 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:46.776661 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.60:35033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 10:44:46.777814 14021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:46.777459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:35033 (local address 127.0.0.1:46944)
0925 10:44:46.777599 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:46.777679 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.60:35033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 10:44:46.778033 14018 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.60:35033: Network error: Client connection negotiation failed: client connection to 127.23.100.60:35033: connect: Connection refused (error 111)
I20260925 10:44:46.833618 13944 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.61" port: 33291 } attempt: 5
Sep 25 10:44:46 dist-test-slave-dxdd krb5kdc[13864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333086, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.62@KRBTEST.COM for kudu/127.23.100.61@KRBTEST.COM
I20260925 10:44:46.846614 14022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:46.834524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38482 (local address 127.23.100.61:33291)
0925 10:44:46.834735 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:46.834741 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:46.834762 (+ 21us) server_negotiation.cc:407] Connection header received
0925 10:44:46.834921 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:46.834932 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:46.834993 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:46.835104 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:46.835869 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:46.836574 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:46.837717 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:46.838157 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:46.838288 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:46.841565 (+ 3277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:46.841570 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:46.841573 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:44:46.841605 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:46.844180 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:46.844780 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:46.844787 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:46.844791 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:46.844913 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:46.845263 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:46.845269 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:46.845273 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:46.845540 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:46.845688 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:46.846052 (+ 364us) server_negotiation.cc:299] Negotiation successful
0925 10:44:46.846258 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 10:44:46.846642 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:46.834410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.61:33291 (local address 127.0.0.1:38482)
0925 10:44:46.834602 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:46.834624 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 10:44:46.834767 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:44:46.835160 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:44:46.835166 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:44:46.835551 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:44:46.835712 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:46.835721 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:46.836710 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:46.836717 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:46.837557 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:46.837568 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:46.838334 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:46.838341 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:46.838549 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:46.839202 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:44:46.839220 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:44:46.841378 (+ 2158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:44:46.844318 (+ 2940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:44:46.844324 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:44:46.844328 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:44:46.844664 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:44:46.845019 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:44:46.845024 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:44:46.845028 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:44:46.845159 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:44:46.845653 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:44:46.845657 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:44:46.845808 (+ 151us) client_negotiation.cc:769] Sending connection context
0925 10:44:46.845951 (+ 143us) client_negotiation.cc:240] Negotiation successful
0925 10:44:46.846259 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 10:44:46.847875 13944 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.60" port: 35033 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:44:46.849046 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:46.848690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:35033 (local address 127.0.0.1:46946)
0925 10:44:46.848830 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:46.848909 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.60:35033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 10:44:46.850030 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:46.849703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:35033 (local address 127.0.0.1:46958)
0925 10:44:46.849826 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:46.849940 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.60:35033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 10:44:46.850293 13944 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.60:35033: Network error: Client connection negotiation failed: client connection to 127.23.100.60:35033: connect: Connection refused (error 111)
I20260925 10:44:46.869263 14018 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.60" port: 35033 } attempt: 3
I20260925 10:44:46.870285 14021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:46.869859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:35033 (local address 127.0.0.1:46968)
0925 10:44:46.870050 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:46.870136 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.60:35033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 10:44:46.871479 14021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:46.870941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:35033 (local address 127.0.0.1:46970)
0925 10:44:46.871233 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:46.871342 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.60:35033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185}
W20260925 10:44:46.871789 14018 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.60:35033: Network error: Client connection negotiation failed: client connection to 127.23.100.60:35033: connect: Connection refused (error 111)
I20260925 10:44:46.901549 13944 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.60" port: 35033 } attempt: 1
I20260925 10:44:46.902645 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:46.902190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:35033 (local address 127.0.0.1:46982)
0925 10:44:46.902411 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:46.902498 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.60:35033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 10:44:46.903556 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:46.903248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:35033 (local address 127.0.0.1:46988)
0925 10:44:46.903382 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:46.903466 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.60:35033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 10:44:46.903764 13944 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.60:35033: Network error: Client connection negotiation failed: client connection to 127.23.100.60:35033: connect: Connection refused (error 111)
I20260925 10:44:46.971940 13944 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.60" port: 35033 } attempt: 2
I20260925 10:44:46.973091 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:46.972618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:35033 (local address 127.0.0.1:47000)
0925 10:44:46.972827 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:46.972946 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.60:35033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 10:44:46.974064 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:46.973746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:35033 (local address 127.0.0.1:47014)
0925 10:44:46.973883 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:46.973964 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.60:35033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 10:44:46.974292 13944 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.60:35033: Network error: Client connection negotiation failed: client connection to 127.23.100.60:35033: connect: Connection refused (error 111)
W20260925 10:44:46.978596 14025 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 10:44:46.978956 14025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:44:46.979058 14025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:44:46.979190 14025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:44:46.986197 14025 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:44:46.986269 14025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:44:46.986433 14025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:44:46.986557 14025 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:44:46.986640 14025 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:44:46.996160 14025 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:38949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.62:40919,127.23.100.61:33291,127.23.100.60:35033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.60.keytab
--principal=kudu/127.23.100.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.60:35033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.100.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:44:46.997356 14025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:44:46.998557 14025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:44:47.009428 14033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:47.009425 14031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:47.009431 14030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:44:47.010046 14025 server_base.cc:1034] running on GCE node
I20260925 10:44:47.010771 14025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:44:47.011947 14025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:44:47.013159 14025 hybrid_clock.cc:648] HybridClock initialized: now 1790333087013093 us; error 50 us; skew 500 ppm
I20260925 10:44:47.013473 14025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:44:47.014992 14018 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.60" port: 35033 } attempt: 4
Sep 25 10:44:47 dist-test-slave-dxdd krb5kdc[13864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333087, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:44:47.015949 14021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:47.015500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:35033 (local address 127.0.0.1:47028)
0925 10:44:47.015734 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:47.015808 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.60:35033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 10:44:47.016616 14025 init.cc:377] Logged in from keytab as kudu/127.23.100.60@KRBTEST.COM (short username kudu)
I20260925 10:44:47.016937 14021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:47.016567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:35033 (local address 127.0.0.1:47034)
0925 10:44:47.016706 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:47.016784 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.60:35033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 10:44:47.017192 14018 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.60:35033: Network error: Client connection negotiation failed: client connection to 127.23.100.60:35033: connect: Connection refused (error 111)
I20260925 10:44:47.018199 14025 webserver.cc:460] Webserver started at http://127.23.100.60:41739/ using document root <none> and password file <none>
I20260925 10:44:47.018903 14025 fs_manager.cc:359] Metadata directory not provided
I20260925 10:44:47.019032 14025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:44:47.019337 14025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:44:47.022374 14025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/minicluster-data/master-2/data/instance:
uuid: "fad1b3adb36d4d448648cc36f123a93c"
format_stamp: "Formatted at 2026-09-25 10:44:47 on dist-test-slave-dxdd"
server_key: "bf63809b4475297c0a1ccfed58efa50c"
server_key_iv: "4a02f143d7154994ade98047b0fc3d8f"
server_key_version: "encryptionkey@0"
I20260925 10:44:47.023146 14025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/minicluster-data/master-2/wal/instance:
uuid: "fad1b3adb36d4d448648cc36f123a93c"
format_stamp: "Formatted at 2026-09-25 10:44:47 on dist-test-slave-dxdd"
server_key: "bf63809b4475297c0a1ccfed58efa50c"
server_key_iv: "4a02f143d7154994ade98047b0fc3d8f"
server_key_version: "encryptionkey@0"
I20260925 10:44:47.028036 14025 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260925 10:44:47.031279 14040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:44:47.032435 14025 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 10:44:47.032608 14025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/minicluster-data/master-2/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/minicluster-data/master-2/wal
uuid: "fad1b3adb36d4d448648cc36f123a93c"
format_stamp: "Formatted at 2026-09-25 10:44:47 on dist-test-slave-dxdd"
server_key: "bf63809b4475297c0a1ccfed58efa50c"
server_key_iv: "4a02f143d7154994ade98047b0fc3d8f"
server_key_version: "encryptionkey@0"
I20260925 10:44:47.032815 14025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:44:47.052570 14025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:44:47.056885 14025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:44:47.057086 14025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:44:47.073812 14025 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.60:35033
I20260925 10:44:47.073829 14091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.60:35033 every 8 connection(s)
I20260925 10:44:47.075106 14025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/minicluster-data/master-2/data/info.pb
I20260925 10:44:47.076258 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 14025
I20260925 10:44:47.076438 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790332740062128-23952-0/minicluster-data/master-2/wal/instance
I20260925 10:44:47.076856 23952 external_mini_cluster.cc:1428] Setting key 9549aab16e5f03562036e5c772c58f26
I20260925 10:44:47.079191 14092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:44:47.080575 13944 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.60" port: 35033 } attempt: 3
I20260925 10:44:47.087997 14092 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.62" port: 40919 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:44:47 dist-test-slave-dxdd krb5kdc[13864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333086, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.62@KRBTEST.COM for kudu/127.23.100.60@KRBTEST.COM
Sep 25 10:44:47 dist-test-slave-dxdd krb5kdc[13864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.60@KRBTEST.COM
I20260925 10:44:47.099680 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:47.081166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:35033 (local address 127.0.0.1:47050)
0925 10:44:47.081366 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:47.081393 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 10:44:47.081502 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:44:47.082553 (+ 1051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:44:47.082561 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:44:47.083035 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:44:47.083248 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:47.083260 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:47.085287 (+ 2027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:47.085292 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:47.086769 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:47.086787 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:47.087901 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:47.087908 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:47.088108 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:47.088768 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:44:47.088791 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:44:47.091627 (+ 2836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:44:47.096385 (+ 4758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:44:47.096390 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:44:47.096395 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:44:47.096916 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:44:47.097406 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:44:47.097416 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:44:47.097421 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:44:47.097592 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:44:47.098643 (+ 1051us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:44:47.098648 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:44:47.098855 (+ 207us) client_negotiation.cc:769] Sending connection context
0925 10:44:47.099022 (+ 167us) client_negotiation.cc:240] Negotiation successful
0925 10:44:47.099277 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
Sep 25 10:44:47 dist-test-slave-dxdd krb5kdc[13864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333087, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.60@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:44:47.101025 14095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:47.078625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47042 (local address 127.23.100.60:35033)
0925 10:44:47.079448 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:47.079460 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:47.079501 (+ 41us) server_negotiation.cc:407] Connection header received
0925 10:44:47.081161 (+ 1660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:47.081186 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:47.081674 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:47.082171 (+ 497us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:47.083562 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:47.085011 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:47.086869 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:47.087573 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:47.087776 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:47.093657 (+ 5881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:47.093663 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:47.093667 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:44:47.093706 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:47.096942 (+ 3236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:47.097766 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:47.097770 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:47.097774 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:47.097980 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:47.098444 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:47.098452 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:47.098469 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:47.098803 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:47.099962 (+ 1159us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:47.100098 (+ 136us) server_negotiation.cc:299] Negotiation successful
0925 10:44:47.100220 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":643,"spinlock_wait_cycles":17152,"thread_start_us":362,"threads_started":1}
I20260925 10:44:47.101068 14096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:47.081493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47050 (local address 127.23.100.60:35033)
0925 10:44:47.082181 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:47.082186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:47.082214 (+ 28us) server_negotiation.cc:407] Connection header received
0925 10:44:47.082263 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:47.082269 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:47.082356 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:47.083401 (+ 1045us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:47.083657 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:47.085079 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:47.086914 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:47.087711 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:47.088437 (+ 726us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:47.091929 (+ 3492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:47.091942 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:47.091960 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:44:47.092004 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:47.096220 (+ 4216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:47.097073 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:47.097080 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:47.097088 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:47.097279 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:47.097744 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:47.097751 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:47.097756 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:47.098407 (+ 651us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:47.098579 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:47.099268 (+ 689us) server_negotiation.cc:299] Negotiation successful
0925 10:44:47.099858 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":546,"thread_start_us":392,"threads_started":1}
I20260925 10:44:47.107934 14097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:47.090937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:40919 (local address 127.0.0.1:60916)
0925 10:44:47.091553 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:47.091584 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 10:44:47.091754 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:44:47.092118 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:44:47.092125 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:44:47.092910 (+ 785us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:44:47.093319 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:47.093332 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:47.094758 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:47.094763 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:47.096691 (+ 1928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:47.096701 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:47.097679 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:47.097687 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:47.097906 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:47.098573 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:44:47.098595 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:44:47.101166 (+ 2571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:44:47.105170 (+ 4004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:44:47.105183 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:44:47.105189 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:44:47.105664 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:44:47.106149 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:44:47.106153 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:44:47.106155 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:44:47.106273 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:44:47.106916 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:44:47.106923 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:44:47.107089 (+ 166us) client_negotiation.cc:769] Sending connection context
0925 10:44:47.107277 (+ 188us) client_negotiation.cc:240] Negotiation successful
0925 10:44:47.107544 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":21,"thread_start_us":303,"threads_started":1}
I20260925 10:44:47.107999 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:47.090866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60916 (local address 127.23.100.62:40919)
0925 10:44:47.091010 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:47.091014 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:47.091675 (+ 661us) server_negotiation.cc:407] Connection header received
0925 10:44:47.091884 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:47.091889 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:47.091956 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:47.092072 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:47.093479 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:47.094567 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:47.096825 (+ 2258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:47.097475 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:47.097592 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:47.101374 (+ 3782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:47.101380 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:47.101386 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:44:47.101419 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:47.105022 (+ 3603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:47.105793 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:47.105799 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:47.105802 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:47.105987 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:47.106481 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:47.106486 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:47.106490 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:47.106798 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:47.106946 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:47.107413 (+ 467us) server_negotiation.cc:299] Negotiation successful
0925 10:44:47.107609 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":34}
I20260925 10:44:47.108618 13944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73d518b5b1974a44add4bea797e5d9f5: Bootstrap starting.
I20260925 10:44:47.109360 14092 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.61" port: 33291 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:44:47.112562 13944 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73d518b5b1974a44add4bea797e5d9f5: Neither blocks nor log segments found. Creating new log.
Sep 25 10:44:47 dist-test-slave-dxdd krb5kdc[13864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:44:47.113788 13944 log.cc:824] T 00000000000000000000000000000000 P 73d518b5b1974a44add4bea797e5d9f5: Log is configured to *not* fsync() on all Append() calls
Sep 25 10:44:47 dist-test-slave-dxdd krb5kdc[13864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333087, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.60@KRBTEST.COM for kudu/127.23.100.61@KRBTEST.COM
I20260925 10:44:47.116962 13944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73d518b5b1974a44add4bea797e5d9f5: No bootstrap required, opened a new log
I20260925 10:44:47.122429 14098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:47.103957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60932 (local address 127.23.100.62:40919)
0925 10:44:47.104550 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:47.104554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:47.104571 (+ 17us) server_negotiation.cc:407] Connection header received
0925 10:44:47.104603 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:47.104606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:47.104666 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:47.104776 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:47.106083 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:47.107440 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:47.109197 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:47.109798 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:47.110495 (+ 697us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:47.114905 (+ 4410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:47.114914 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:47.114925 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:44:47.114970 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:47.118291 (+ 3321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:47.119834 (+ 1543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:47.119841 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:47.119848 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:47.120029 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:47.120480 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:47.120487 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:47.120496 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:47.120843 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:47.121817 (+ 974us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:47.121908 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 10:44:47.122058 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":474,"thread_start_us":283,"threads_started":1}
I20260925 10:44:47.123981 14097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:47.109947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.61:33291 (local address 127.0.0.1:38496)
0925 10:44:47.110163 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:47.110178 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:44:47.110264 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:44:47.110776 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:44:47.110779 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:44:47.111051 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:44:47.111157 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:47.111164 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:47.112094 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:47.112099 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:47.113031 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:47.113062 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:47.114063 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:47.114069 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:47.114268 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:47.114841 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:44:47.114862 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:44:47.117288 (+ 2426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:44:47.121004 (+ 3716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:44:47.121009 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:44:47.121013 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:44:47.121510 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:44:47.121970 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:44:47.121974 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:44:47.121978 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:44:47.122136 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:44:47.123003 (+ 867us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:44:47.123009 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:44:47.123159 (+ 150us) client_negotiation.cc:769] Sending connection context
0925 10:44:47.123341 (+ 182us) client_negotiation.cc:240] Negotiation successful
0925 10:44:47.123622 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121}
I20260925 10:44:47.124136 14022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:47.110026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38496 (local address 127.23.100.61:33291)
0925 10:44:47.110191 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:47.110199 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:47.110219 (+ 20us) server_negotiation.cc:407] Connection header received
0925 10:44:47.110419 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:47.110428 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:47.110513 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:47.110605 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:47.111294 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:47.111949 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:47.113172 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:47.113780 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:47.113928 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:47.117499 (+ 3571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:47.117505 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:47.117510 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:44:47.117548 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:47.120876 (+ 3328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:47.121660 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:47.121665 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:47.121669 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:47.121815 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:47.122269 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:47.122274 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:47.122277 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:47.122838 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:47.123045 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:47.123461 (+ 416us) server_negotiation.cc:299] Negotiation successful
0925 10:44:47.123688 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 10:44:47.125466 14092 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.60" port: 35033 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:44:47.126689 13944 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73d518b5b1974a44add4bea797e5d9f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d518b5b1974a44add4bea797e5d9f5" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 40919 } } peers { permanent_uuid: "b7672de60ddc48a087db664d20970e22" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 33291 } } peers { permanent_uuid: "fad1b3adb36d4d448648cc36f123a93c" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 35033 } }
I20260925 10:44:47.126991 13944 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73d518b5b1974a44add4bea797e5d9f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73d518b5b1974a44add4bea797e5d9f5, State: Initialized, Role: FOLLOWER
I20260925 10:44:47.127359 13944 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73d518b5b1974a44add4bea797e5d9f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d518b5b1974a44add4bea797e5d9f5" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 40919 } } peers { permanent_uuid: "b7672de60ddc48a087db664d20970e22" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 33291 } } peers { permanent_uuid: "fad1b3adb36d4d448648cc36f123a93c" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 35033 } }
I20260925 10:44:47.128135 14101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73d518b5b1974a44add4bea797e5d9f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d518b5b1974a44add4bea797e5d9f5" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 40919 } } peers { permanent_uuid: "b7672de60ddc48a087db664d20970e22" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 33291 } } peers { permanent_uuid: "fad1b3adb36d4d448648cc36f123a93c" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 35033 } } }
I20260925 10:44:47.128502 14101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73d518b5b1974a44add4bea797e5d9f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:44:47.129177 13944 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73d518b5b1974a44add4bea797e5d9f5 [sys.catalog]: configured and running, proceeding with master startup.
Sep 25 10:44:47 dist-test-slave-dxdd krb5kdc[13864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.61@KRBTEST.COM
Sep 25 10:44:47 dist-test-slave-dxdd krb5kdc[13864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333087, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.60@KRBTEST.COM for kudu/127.23.100.60@KRBTEST.COM
I20260925 10:44:47.143222 14100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:47.124378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38510 (local address 127.23.100.61:33291)
0925 10:44:47.124948 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:47.124956 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:47.124979 (+ 23us) server_negotiation.cc:407] Connection header received
0925 10:44:47.125025 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:47.125030 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:47.125091 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:47.125219 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:47.126795 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:47.127950 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:47.129628 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:47.130267 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:47.130935 (+ 668us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:47.135703 (+ 4768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:47.135709 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:47.135714 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:44:47.135749 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:47.139346 (+ 3597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:47.140407 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:47.140412 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:47.140415 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:47.140532 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:47.140973 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:47.140977 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:47.140979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:47.141279 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:47.142495 (+ 1216us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:47.142612 (+ 117us) server_negotiation.cc:299] Negotiation successful
0925 10:44:47.142830 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":256,"threads_started":1}
I20260925 10:44:47.144311 14095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:47.126390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47054 (local address 127.23.100.60:35033)
0925 10:44:47.126551 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:47.126557 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:47.126909 (+ 352us) server_negotiation.cc:407] Connection header received
0925 10:44:47.127043 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:47.127047 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:47.127151 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:47.127248 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:47.128214 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:47.129383 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:47.131080 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:47.131740 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:47.131888 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:47.136220 (+ 4332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:47.136228 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:47.136235 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:44:47.136274 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:47.140357 (+ 4083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:47.141205 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:47.141210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:47.141213 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:47.141411 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:47.141838 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:47.141846 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:47.141851 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:47.142274 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:47.142496 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:47.143753 (+ 1257us) server_negotiation.cc:299] Negotiation successful
0925 10:44:47.143992 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 10:44:47.144927 14097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:47.126477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:35033 (local address 127.0.0.1:47054)
0925 10:44:47.126671 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:47.126683 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:44:47.126807 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:44:47.127375 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:44:47.127380 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:44:47.127778 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:44:47.127979 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:47.127991 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:47.129520 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:47.129526 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:47.130936 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:47.130949 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:47.132056 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:47.132064 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:47.132275 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:47.132823 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:44:47.132853 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:44:47.136012 (+ 3159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:44:47.140515 (+ 4503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:44:47.140521 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:44:47.140527 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:44:47.141074 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:44:47.141534 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:44:47.141540 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:44:47.141546 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:44:47.141716 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:44:47.142552 (+ 836us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:44:47.142559 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:44:47.142738 (+ 179us) client_negotiation.cc:769] Sending connection context
0925 10:44:47.142913 (+ 175us) client_negotiation.cc:240] Negotiation successful
0925 10:44:47.143255 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125}
I20260925 10:44:47.149343 14092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fad1b3adb36d4d448648cc36f123a93c: Bootstrap starting.
I20260925 10:44:47.151837 23952 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:44:47.152829 14092 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fad1b3adb36d4d448648cc36f123a93c: Neither blocks nor log segments found. Creating new log.
I20260925 10:44:47.153978 14092 log.cc:824] T 00000000000000000000000000000000 P fad1b3adb36d4d448648cc36f123a93c: Log is configured to *not* fsync() on all Append() calls
I20260925 10:44:47.155283 13919 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "73d518b5b1974a44add4bea797e5d9f5"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60932
I20260925 10:44:47.155754 13919 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73d518b5b1974a44add4bea797e5d9f5 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 10:44:47.155973 13919 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73d518b5b1974a44add4bea797e5d9f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:44:47.157226 14092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fad1b3adb36d4d448648cc36f123a93c: No bootstrap required, opened a new log
I20260925 10:44:47.158258 13919 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73d518b5b1974a44add4bea797e5d9f5 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d518b5b1974a44add4bea797e5d9f5" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 40919 } } peers { permanent_uuid: "b7672de60ddc48a087db664d20970e22" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 33291 } } peers { permanent_uuid: "fad1b3adb36d4d448648cc36f123a93c" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 35033 } }
I20260925 10:44:47.160281 13919 leader_election.cc:288] T 00000000000000000000000000000000 P 73d518b5b1974a44add4bea797e5d9f5 [CANDIDATE]: Term 1 election: Requested vote from peers b7672de60ddc48a087db664d20970e22 (127.23.100.61:33291), fad1b3adb36d4d448648cc36f123a93c (127.23.100.60:35033)
W20260925 10:44:47.167263 14113 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 73d518b5b1974a44add4bea797e5d9f5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:44:47.167385 14113 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:44:47.167632 14092 raft_consensus.cc:348] T 00000000000000000000000000000000 P fad1b3adb36d4d448648cc36f123a93c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d518b5b1974a44add4bea797e5d9f5" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 40919 } } peers { permanent_uuid: "b7672de60ddc48a087db664d20970e22" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 33291 } } peers { permanent_uuid: "fad1b3adb36d4d448648cc36f123a93c" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 35033 } }
I20260925 10:44:47.168008 14092 raft_consensus.cc:729] T 00000000000000000000000000000000 P fad1b3adb36d4d448648cc36f123a93c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fad1b3adb36d4d448648cc36f123a93c, State: Initialized, Role: FOLLOWER
I20260925 10:44:47.168460 14092 consensus_queue.cc:260] T 00000000000000000000000000000000 P fad1b3adb36d4d448648cc36f123a93c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d518b5b1974a44add4bea797e5d9f5" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 40919 } } peers { permanent_uuid: "b7672de60ddc48a087db664d20970e22" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 33291 } } peers { permanent_uuid: "fad1b3adb36d4d448648cc36f123a93c" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 35033 } }
I20260925 10:44:47.169315 14117 sys_catalog.cc:455] T 00000000000000000000000000000000 P fad1b3adb36d4d448648cc36f123a93c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d518b5b1974a44add4bea797e5d9f5" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 40919 } } peers { permanent_uuid: "b7672de60ddc48a087db664d20970e22" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 33291 } } peers { permanent_uuid: "fad1b3adb36d4d448648cc36f123a93c" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 35033 } } }
I20260925 10:44:47.169701 14117 sys_catalog.cc:458] T 00000000000000000000000000000000 P fad1b3adb36d4d448648cc36f123a93c [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:44:47.170424 14092 sys_catalog.cc:564] T 00000000000000000000000000000000 P fad1b3adb36d4d448648cc36f123a93c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:44:47.170586 14067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "73d518b5b1974a44add4bea797e5d9f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fad1b3adb36d4d448648cc36f123a93c"
I20260925 10:44:47.171031 14067 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fad1b3adb36d4d448648cc36f123a93c [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:44:47.175304 14067 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fad1b3adb36d4d448648cc36f123a93c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73d518b5b1974a44add4bea797e5d9f5 in term 1.
I20260925 10:44:47.176039 13896 leader_election.cc:302] T 00000000000000000000000000000000 P 73d518b5b1974a44add4bea797e5d9f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73d518b5b1974a44add4bea797e5d9f5, fad1b3adb36d4d448648cc36f123a93c; no voters:
I20260925 10:44:47.164768 13877 master_runner.cc:186] Error getting master registration for 127.23.100.61:33291: OK, instance_id { permanent_uuid: "b7672de60ddc48a087db664d20970e22" instance_seqno: 1790333086597104 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 10:44:47.176565 13877 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 10:44:47.176597 14101 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73d518b5b1974a44add4bea797e5d9f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:44:47.188445 14101 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73d518b5b1974a44add4bea797e5d9f5 [term 1 LEADER]: Becoming Leader. State: Replica: 73d518b5b1974a44add4bea797e5d9f5, State: Running, Role: LEADER
I20260925 10:44:47.189229 14101 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73d518b5b1974a44add4bea797e5d9f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d518b5b1974a44add4bea797e5d9f5" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 40919 } } peers { permanent_uuid: "b7672de60ddc48a087db664d20970e22" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 33291 } } peers { permanent_uuid: "fad1b3adb36d4d448648cc36f123a93c" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 35033 } }
I20260925 10:44:47.227861 14100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:47.186751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38518 (local address 127.23.100.61:33291)
0925 10:44:47.186926 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:47.186932 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:47.205952 (+ 19020us) server_negotiation.cc:407] Connection header received
0925 10:44:47.206380 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:47.206390 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:47.206495 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:47.206579 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:47.209799 (+ 3220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:47.210880 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:47.214044 (+ 3164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:47.214740 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:47.214837 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:47.220025 (+ 5188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:47.220032 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:47.220036 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:44:47.220077 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:47.223844 (+ 3767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:47.225043 (+ 1199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:47.225048 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:47.225051 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:47.225178 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:47.225770 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:47.225777 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:47.225782 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:47.226122 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:47.226378 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:47.227213 (+ 835us) server_negotiation.cc:299] Negotiation successful
0925 10:44:47.227475 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 10:44:47.234834 14098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:47.184759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60940 (local address 127.23.100.62:40919)
0925 10:44:47.184977 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:47.184983 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:47.190872 (+ 5889us) server_negotiation.cc:407] Connection header received
0925 10:44:47.197969 (+ 7097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:47.197977 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:47.198072 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:47.198155 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:47.207331 (+ 9176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:47.208412 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:47.217698 (+ 9286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:47.218499 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:47.218609 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:47.227038 (+ 8429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:47.227044 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:47.227050 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:44:47.227090 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:47.230842 (+ 3752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:47.231932 (+ 1090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:47.231941 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:47.231947 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:47.232120 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:47.232621 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:47.232627 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:47.232632 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:47.233085 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:47.233373 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:47.234019 (+ 646us) server_negotiation.cc:299] Negotiation successful
0925 10:44:47.234332 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 10:44:47.240635 14121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73d518b5b1974a44add4bea797e5d9f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73d518b5b1974a44add4bea797e5d9f5. Latest consensus state: current_term: 1 leader_uuid: "73d518b5b1974a44add4bea797e5d9f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d518b5b1974a44add4bea797e5d9f5" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 40919 } } peers { permanent_uuid: "b7672de60ddc48a087db664d20970e22" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 33291 } } peers { permanent_uuid: "fad1b3adb36d4d448648cc36f123a93c" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 35033 } } }
I20260925 10:44:47.241115 14121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73d518b5b1974a44add4bea797e5d9f5 [sys.catalog]: This master's current role is: LEADER
I20260925 10:44:47.252210 14131 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:44:47.254434 14131 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:44:47.263619 14095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:47.192050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47062 (local address 127.23.100.60:35033)
0925 10:44:47.192297 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:47.192305 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:47.207002 (+ 14697us) server_negotiation.cc:407] Connection header received
0925 10:44:47.210665 (+ 3663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:47.210671 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:47.210741 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:47.210827 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:47.219672 (+ 8845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:47.220781 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:47.224998 (+ 4217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:47.225619 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:47.225719 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:47.244328 (+ 18609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:47.244339 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:47.244347 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:44:47.244403 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:47.249210 (+ 4807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:47.251735 (+ 2525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:47.251895 (+ 160us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:47.251902 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:47.252173 (+ 271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:47.252841 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:47.252848 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:47.252875 (+ 27us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:47.253548 (+ 673us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:47.253948 (+ 400us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:47.263053 (+ 9105us) server_negotiation.cc:299] Negotiation successful
0925 10:44:47.263303 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 10:44:47.266520 14067 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fad1b3adb36d4d448648cc36f123a93c [term 1 FOLLOWER]: Refusing update from remote peer 73d518b5b1974a44add4bea797e5d9f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:44:47.267307 14121 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 73d518b5b1974a44add4bea797e5d9f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fad1b3adb36d4d448648cc36f123a93c" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 35033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:44:47.277204 14117 sys_catalog.cc:455] T 00000000000000000000000000000000 P fad1b3adb36d4d448648cc36f123a93c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73d518b5b1974a44add4bea797e5d9f5. Latest consensus state: current_term: 1 leader_uuid: "73d518b5b1974a44add4bea797e5d9f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d518b5b1974a44add4bea797e5d9f5" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 40919 } } peers { permanent_uuid: "b7672de60ddc48a087db664d20970e22" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 33291 } } peers { permanent_uuid: "fad1b3adb36d4d448648cc36f123a93c" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 35033 } } }
I20260925 10:44:47.277480 14117 sys_catalog.cc:458] T 00000000000000000000000000000000 P fad1b3adb36d4d448648cc36f123a93c [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:44:47.279464 14133 mvcc.cc:205] Tried to move back new op lower bound from 7333204325430808576 to 7333204325133750272. Current Snapshot: MvccSnapshot[applied={T|T < 7333204325430808576}]
I20260925 10:44:47.295575 14117 sys_catalog.cc:455] T 00000000000000000000000000000000 P fad1b3adb36d4d448648cc36f123a93c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "73d518b5b1974a44add4bea797e5d9f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d518b5b1974a44add4bea797e5d9f5" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 40919 } } peers { permanent_uuid: "b7672de60ddc48a087db664d20970e22" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 33291 } } peers { permanent_uuid: "fad1b3adb36d4d448648cc36f123a93c" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 35033 } } }
I20260925 10:44:47.295905 14117 sys_catalog.cc:458] T 00000000000000000000000000000000 P fad1b3adb36d4d448648cc36f123a93c [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:44:47.302055 14121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73d518b5b1974a44add4bea797e5d9f5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "73d518b5b1974a44add4bea797e5d9f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d518b5b1974a44add4bea797e5d9f5" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 40919 } } peers { permanent_uuid: "b7672de60ddc48a087db664d20970e22" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 33291 } } peers { permanent_uuid: "fad1b3adb36d4d448648cc36f123a93c" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 35033 } } }
I20260925 10:44:47.308166 14121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73d518b5b1974a44add4bea797e5d9f5 [sys.catalog]: This master's current role is: LEADER
I20260925 10:44:47.316350 14018 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.60" port: 35033 } attempt: 5
Sep 25 10:44:47 dist-test-slave-dxdd krb5kdc[13864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333086, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.61@KRBTEST.COM for kudu/127.23.100.60@KRBTEST.COM
I20260925 10:44:47.339332 14131 catalog_manager.cc:1347] Generated new cluster ID: 40f8f117dbae4eae8ea4dc3c3fb957c1
I20260925 10:44:47.339452 14131 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:44:47.312186 14025 master_runner.cc:186] Error getting master registration for 127.23.100.61:33291: OK, instance_id { permanent_uuid: "b7672de60ddc48a087db664d20970e22" instance_seqno: 1790333086597104 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 10:44:47.340893 14025 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 10:44:47.341281 14139 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fad1b3adb36d4d448648cc36f123a93c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:44:47.341401 14139 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:44:47.344741 14095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:47.317635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47064 (local address 127.23.100.60:35033)
0925 10:44:47.317792 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:47.317798 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:47.317816 (+ 18us) server_negotiation.cc:407] Connection header received
0925 10:44:47.317869 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:47.317873 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:47.317983 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:47.318074 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:47.319309 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:47.320450 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:47.322317 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:47.323031 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:47.323157 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:47.329525 (+ 6368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:47.329532 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:47.329536 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:44:47.329574 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:47.335313 (+ 5739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:47.341806 (+ 6493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:47.341812 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:47.341816 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:47.341976 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:47.342756 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:47.342761 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:47.342765 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:47.343154 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:47.343288 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:47.344142 (+ 854us) server_negotiation.cc:299] Negotiation successful
0925 10:44:47.344378 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 10:44:47.346365 14021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:47.316956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:35033 (local address 127.0.0.1:47064)
0925 10:44:47.317282 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:47.317312 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 10:44:47.317412 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:44:47.318281 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:44:47.318295 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:44:47.318878 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:44:47.319092 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:47.319106 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:47.320660 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:47.320666 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:47.322076 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:47.322091 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:47.323380 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:47.323385 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:47.323592 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:47.324369 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:44:47.324395 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:44:47.329228 (+ 4833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:44:47.339744 (+ 10516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:44:47.339752 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:44:47.339758 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:44:47.341635 (+ 1877us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:44:47.342176 (+ 541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:44:47.342181 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:44:47.342185 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:44:47.342546 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:44:47.343420 (+ 874us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:44:47.343427 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:44:47.343722 (+ 295us) client_negotiation.cc:769] Sending connection context
0925 10:44:47.345202 (+ 1480us) client_negotiation.cc:240] Negotiation successful
0925 10:44:47.345723 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205}
I20260925 10:44:47.352119 14018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7672de60ddc48a087db664d20970e22: Bootstrap starting.
I20260925 10:44:47.358661 14018 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7672de60ddc48a087db664d20970e22: Neither blocks nor log segments found. Creating new log.
I20260925 10:44:47.359428 14131 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:44:47.360067 14018 log.cc:824] T 00000000000000000000000000000000 P b7672de60ddc48a087db664d20970e22: Log is configured to *not* fsync() on all Append() calls
I20260925 10:44:47.362766 14131 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:44:47.379379 14018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7672de60ddc48a087db664d20970e22: No bootstrap required, opened a new log
I20260925 10:44:47.387072 14131 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73d518b5b1974a44add4bea797e5d9f5: Generated new TSK 0
I20260925 10:44:47.387995 14131 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:44:47.390704 14018 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7672de60ddc48a087db664d20970e22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d518b5b1974a44add4bea797e5d9f5" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 40919 } } peers { permanent_uuid: "b7672de60ddc48a087db664d20970e22" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 33291 } } peers { permanent_uuid: "fad1b3adb36d4d448648cc36f123a93c" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 35033 } }
I20260925 10:44:47.391180 14018 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7672de60ddc48a087db664d20970e22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7672de60ddc48a087db664d20970e22, State: Initialized, Role: FOLLOWER
I20260925 10:44:47.391779 14018 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7672de60ddc48a087db664d20970e22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d518b5b1974a44add4bea797e5d9f5" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 40919 } } peers { permanent_uuid: "b7672de60ddc48a087db664d20970e22" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 33291 } } peers { permanent_uuid: "fad1b3adb36d4d448648cc36f123a93c" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 35033 } }
I20260925 10:44:47.398980 14145 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7672de60ddc48a087db664d20970e22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d518b5b1974a44add4bea797e5d9f5" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 40919 } } peers { permanent_uuid: "b7672de60ddc48a087db664d20970e22" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 33291 } } peers { permanent_uuid: "fad1b3adb36d4d448648cc36f123a93c" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 35033 } } }
I20260925 10:44:47.420933 14145 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7672de60ddc48a087db664d20970e22 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:44:47.420681 14018 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7672de60ddc48a087db664d20970e22 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:44:47.421082 13992 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7672de60ddc48a087db664d20970e22 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:44:47.421231 13993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "73d518b5b1974a44add4bea797e5d9f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b7672de60ddc48a087db664d20970e22"
I20260925 10:44:47.424780 13992 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b7672de60ddc48a087db664d20970e22 [term 1 FOLLOWER]: Refusing update from remote peer 73d518b5b1974a44add4bea797e5d9f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:44:47.425748 14101 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 73d518b5b1974a44add4bea797e5d9f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7672de60ddc48a087db664d20970e22" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 33291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:44:47.429641 13993 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b7672de60ddc48a087db664d20970e22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73d518b5b1974a44add4bea797e5d9f5 in term 1.
I20260925 10:44:47.503104 14154 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7672de60ddc48a087db664d20970e22 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "73d518b5b1974a44add4bea797e5d9f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d518b5b1974a44add4bea797e5d9f5" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 40919 } } peers { permanent_uuid: "b7672de60ddc48a087db664d20970e22" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 33291 } } peers { permanent_uuid: "fad1b3adb36d4d448648cc36f123a93c" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 35033 } } }
I20260925 10:44:47.503540 14154 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7672de60ddc48a087db664d20970e22 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:44:47.503885 14145 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7672de60ddc48a087db664d20970e22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73d518b5b1974a44add4bea797e5d9f5. Latest consensus state: current_term: 1 leader_uuid: "73d518b5b1974a44add4bea797e5d9f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d518b5b1974a44add4bea797e5d9f5" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 40919 } } peers { permanent_uuid: "b7672de60ddc48a087db664d20970e22" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 33291 } } peers { permanent_uuid: "fad1b3adb36d4d448648cc36f123a93c" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 35033 } } }
I20260925 10:44:47.504128 14145 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7672de60ddc48a087db664d20970e22 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:44:47.505095 14163 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260925 10:44:47.511780 23952 mini_kdc.cc:348] Time spent kdestroy: real 0.105s user 0.000s sys 0.021s
W20260925 10:44:47.538286 14100 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:44:47.520821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38524 (local address 127.23.100.61:33291)
0925 10:44:47.520991 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:47.520999 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:47.535489 (+ 14490us) server_negotiation.cc:407] Connection header received
0925 10:44:47.536016 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:47.536022 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:47.536121 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:47.536194 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:47.538008 (+ 1814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38524: BlockingRecv error: recv got EOF from 127.0.0.1:38524 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260925 10:44:47.539960 14098 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:44:47.520837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60944 (local address 127.23.100.62:40919)
0925 10:44:47.521061 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:47.521069 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:47.534860 (+ 13791us) server_negotiation.cc:407] Connection header received
0925 10:44:47.536493 (+ 1633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:47.536499 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:47.536604 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:47.536674 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:47.539660 (+ 2986us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60944: BlockingRecv error: recv got EOF from 127.0.0.1:60944 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260925 10:44:47.542610 14095 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:44:47.526947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47070 (local address 127.23.100.60:35033)
0925 10:44:47.527105 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:47.527122 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:47.540653 (+ 13531us) server_negotiation.cc:407] Connection header received
0925 10:44:47.541252 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:47.541258 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:47.541337 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:47.541418 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:47.542331 (+ 913us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47070: BlockingRecv error: recv got EOF from 127.0.0.1:47070 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260925 10:44:47.554260 14100 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:44:47.552206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38530 (local address 127.23.100.61:33291)
0925 10:44:47.552374 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:47.552380 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:47.552846 (+ 466us) server_negotiation.cc:407] Connection header received
0925 10:44:47.553070 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:47.553076 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:47.553190 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:47.553295 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:47.554050 (+ 755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38530: BlockingRecv error: recv got EOF from 127.0.0.1:38530 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260925 10:44:47.567196 14100 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:44:47.561913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38534 (local address 127.23.100.61:33291)
0925 10:44:47.562075 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:47.562080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:47.565810 (+ 3730us) server_negotiation.cc:407] Connection header received
0925 10:44:47.565973 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:47.565978 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:47.566074 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:47.566177 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:47.566953 (+ 776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38534: BlockingRecv error: recv got EOF from 127.0.0.1:38534 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260925 10:44:47.574467 14100 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:44:47.572175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38550 (local address 127.23.100.61:33291)
0925 10:44:47.572345 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:47.572350 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:47.572894 (+ 544us) server_negotiation.cc:407] Connection header received
0925 10:44:47.573159 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:47.573166 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:47.573245 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:47.573317 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:47.574221 (+ 904us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38550: BlockingRecv error: recv got EOF from 127.0.0.1:38550 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260925 10:44:47.584224 14100 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:44:47.582228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38562 (local address 127.23.100.61:33291)
0925 10:44:47.582445 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:47.582450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:47.582930 (+ 480us) server_negotiation.cc:407] Connection header received
0925 10:44:47.583099 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:47.583104 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:47.583181 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:47.583279 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:47.584008 (+ 729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38562: BlockingRecv error: recv got EOF from 127.0.0.1:38562 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
W20260925 10:44:47.599583 14100 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:44:47.596100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38572 (local address 127.23.100.61:33291)
0925 10:44:47.596276 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:47.596282 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:47.596780 (+ 498us) server_negotiation.cc:407] Connection header received
0925 10:44:47.596985 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:47.596994 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:47.597087 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:47.597202 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:47.599352 (+ 2150us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38572: BlockingRecv error: recv got EOF from 127.0.0.1:38572 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260925 10:44:47.638615 14100 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:44:47.636208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38576 (local address 127.23.100.61:33291)
0925 10:44:47.636403 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:47.636410 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:47.636972 (+ 562us) server_negotiation.cc:407] Connection header received
0925 10:44:47.637235 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:47.637242 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:47.637339 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:47.637467 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:47.638358 (+ 891us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38576: BlockingRecv error: recv got EOF from 127.0.0.1:38576 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260925 10:44:47.676721 14100 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:44:47.674308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38580 (local address 127.23.100.61:33291)
0925 10:44:47.674521 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:47.674528 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:47.675104 (+ 576us) server_negotiation.cc:407] Connection header received
0925 10:44:47.675315 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:47.675321 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:47.675430 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:47.675550 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:47.676464 (+ 914us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38580: BlockingRecv error: recv got EOF from 127.0.0.1:38580 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260925 10:44:47.747025 14100 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:44:47.744645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38590 (local address 127.23.100.61:33291)
0925 10:44:47.744846 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:47.744855 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:47.745400 (+ 545us) server_negotiation.cc:407] Connection header received
0925 10:44:47.745621 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:47.745627 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:47.745718 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:47.745812 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:47.746765 (+ 953us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38590: BlockingRecv error: recv got EOF from 127.0.0.1:38590 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260925 10:44:47.883894 14100 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:44:47.882008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38592 (local address 127.23.100.61:33291)
0925 10:44:47.882248 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:47.882254 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:47.882459 (+ 205us) server_negotiation.cc:407] Connection header received
0925 10:44:47.882723 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:47.882731 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:47.882865 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:47.882976 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:47.883642 (+ 666us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38592: BlockingRecv error: recv got EOF from 127.0.0.1:38592 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 10:44:48.049475 14101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73d518b5b1974a44add4bea797e5d9f5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "73d518b5b1974a44add4bea797e5d9f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d518b5b1974a44add4bea797e5d9f5" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 40919 } } peers { permanent_uuid: "b7672de60ddc48a087db664d20970e22" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 33291 } } peers { permanent_uuid: "fad1b3adb36d4d448648cc36f123a93c" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 35033 } } }
I20260925 10:44:48.049811 14101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73d518b5b1974a44add4bea797e5d9f5 [sys.catalog]: This master's current role is: LEADER
W20260925 10:44:48.144964 14100 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:44:48.143158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38596 (local address 127.23.100.61:33291)
0925 10:44:48.143309 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:48.143313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:48.143700 (+ 387us) server_negotiation.cc:407] Connection header received
0925 10:44:48.143851 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:48.143856 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:48.143975 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:48.144089 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:48.144741 (+ 652us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38596: BlockingRecv error: recv got EOF from 127.0.0.1:38596 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 10:44:48.343094 14139 catalog_manager.cc:1259] Loaded cluster ID: 40f8f117dbae4eae8ea4dc3c3fb957c1
I20260925 10:44:48.343232 14139 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fad1b3adb36d4d448648cc36f123a93c: loading cluster ID for follower catalog manager: success
I20260925 10:44:48.345707 14139 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fad1b3adb36d4d448648cc36f123a93c: acquiring CA information for follower catalog manager: success
I20260925 10:44:48.347417 14139 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fad1b3adb36d4d448648cc36f123a93c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 10:44:48.509274 14163 catalog_manager.cc:1259] Loaded cluster ID: 40f8f117dbae4eae8ea4dc3c3fb957c1
I20260925 10:44:48.509430 14163 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b7672de60ddc48a087db664d20970e22: loading cluster ID for follower catalog manager: success
I20260925 10:44:48.512136 14163 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b7672de60ddc48a087db664d20970e22: acquiring CA information for follower catalog manager: success
I20260925 10:44:48.513885 14163 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b7672de60ddc48a087db664d20970e22: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 10:44:48.667446 14230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:48.661426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38602 (local address 127.23.100.61:33291)
0925 10:44:48.662235 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:48.662240 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:48.662256 (+ 16us) server_negotiation.cc:407] Connection header received
0925 10:44:48.662365 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:48.662370 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:48.662558 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:48.662692 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:44:48.663140 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:48.663930 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:48.665195 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:48.665658 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:48.665791 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:48.666245 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:44:48.666840 (+ 595us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:44:48.666997 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:48.667095 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 10:44:48.667175 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":494,"threads_started":1}
I20260925 10:44:48.691566 14236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:48.685487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47082 (local address 127.23.100.60:35033)
0925 10:44:48.686098 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:48.686102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:48.686117 (+ 15us) server_negotiation.cc:407] Connection header received
0925 10:44:48.686204 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:48.686208 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:48.686410 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:48.686563 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:44:48.686937 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:48.687583 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:48.689379 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:48.689821 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:48.689931 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:48.690485 (+ 554us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:44:48.690931 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:44:48.691037 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:48.691186 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 10:44:48.691247 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":396,"threads_started":1}
I20260925 10:44:48.711833 14230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:48.698641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38608 (local address 127.23.100.61:33291)
0925 10:44:48.698997 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:48.699006 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:48.703451 (+ 4445us) server_negotiation.cc:407] Connection header received
0925 10:44:48.703941 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:48.703962 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:48.704077 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:48.704153 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:44:48.705019 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:48.706474 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:48.709167 (+ 2693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:48.710072 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:48.710193 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:48.711019 (+ 826us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:44:48.711153 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:44:48.711213 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:48.711484 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 10:44:48.711563 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202}
I20260925 10:44:48.718128 14243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:48.698513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60958 (local address 127.23.100.62:40919)
0925 10:44:48.711569 (+ 13056us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:48.711575 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:48.711592 (+ 17us) server_negotiation.cc:407] Connection header received
0925 10:44:48.711694 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:48.711699 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:48.711932 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:48.712631 (+ 699us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:44:48.712765 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:48.713823 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:48.716090 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:48.716590 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:48.716729 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:48.717172 (+ 443us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:44:48.717468 (+ 296us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:44:48.717586 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:48.717716 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 10:44:48.717813 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12936,"thread_start_us":13559,"threads_started":1}
I20260925 10:44:48.718581 14236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:48.698420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47098 (local address 127.23.100.60:35033)
0925 10:44:48.698811 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:48.698817 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:48.710811 (+ 11994us) server_negotiation.cc:407] Connection header received
0925 10:44:48.710971 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:48.710976 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:48.711055 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:48.711136 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:44:48.711659 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:48.712749 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:48.716356 (+ 3607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:48.717010 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:48.717132 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:48.717713 (+ 581us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:44:48.717830 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:44:48.717918 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:48.718198 (+ 280us) server_negotiation.cc:299] Negotiation successful
0925 10:44:48.718275 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232}
I20260925 10:44:48.723484 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 13877
I20260925 10:44:48.724618 13939 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:44:48.834158 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 13877
I20260925 10:44:48.855871 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 13951
I20260925 10:44:48.856854 14013 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:44:48.959702 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 13951
I20260925 10:44:48.978933 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 14025
I20260925 10:44:48.980007 14087 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:44:49.087175 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 14025
2026-09-25T10:44:49Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4721 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:44:49 dist-test-slave-dxdd krb5kdc[14250](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:44:49 dist-test-slave-dxdd krb5kdc[14250](info): set up 2 sockets
Sep 25 10:44:49 dist-test-slave-dxdd krb5kdc[14250](info): commencing operation
krb5kdc: starting...
W20260925 10:44:50.174973 23952 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/krb5kdc/test-user.keytab.
Sep 25 10:44:50 dist-test-slave-dxdd krb5kdc[14250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:44:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:44:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
Entry for principal kudu/127.23.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab.
I20260925 10:44:50.421068 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:36181
--webserver_interface=127.23.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:43715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.62:36181,127.23.100.61:41659,127.23.100.60:36045
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:44:50.737481 14263 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 10:44:50.737777 14263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:44:50.737840 14263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:44:50.737911 14263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:44:50.744951 14263 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:44:50.745031 14263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:44:50.745085 14263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:44:50.745134 14263 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:44:50.745186 14263 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:44:50.754441 14263 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:43715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.62:36181,127.23.100.61:41659,127.23.100.60:36045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.62.keytab
--principal=kudu/127.23.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.62:36181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:44:50.755589 14263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:44:50.756853 14263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:44:50.768483 14268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:50.768539 14269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:50.768738 14271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:44:50.769299 14263 server_base.cc:1034] running on GCE node
I20260925 10:44:50.769999 14263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:44:50.771355 14263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:44:50.772569 14263 hybrid_clock.cc:648] HybridClock initialized: now 1790333090772517 us; error 37 us; skew 500 ppm
I20260925 10:44:50.772898 14263 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:44:50 dist-test-slave-dxdd krb5kdc[14250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333090, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:44:50.775941 14263 init.cc:377] Logged in from keytab as kudu/127.23.100.62@KRBTEST.COM (short username kudu)
I20260925 10:44:50.777276 14263 webserver.cc:460] Webserver started at http://127.23.100.62:41067/ using document root <none> and password file <none>
I20260925 10:44:50.777956 14263 fs_manager.cc:359] Metadata directory not provided
I20260925 10:44:50.778082 14263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:44:50.778452 14263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:44:50.781333 14263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/minicluster-data/master-0/data/instance:
uuid: "68f57a46decf45fda1b4213afe411d80"
format_stamp: "Formatted at 2026-09-25 10:44:50 on dist-test-slave-dxdd"
server_key: "17536d3ab4e142f0427373af01e1c644"
server_key_iv: "f69a515a5bafeb472460df0f432202e0"
server_key_version: "encryptionkey@0"
I20260925 10:44:50.782124 14263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/minicluster-data/master-0/wal/instance:
uuid: "68f57a46decf45fda1b4213afe411d80"
format_stamp: "Formatted at 2026-09-25 10:44:50 on dist-test-slave-dxdd"
server_key: "17536d3ab4e142f0427373af01e1c644"
server_key_iv: "f69a515a5bafeb472460df0f432202e0"
server_key_version: "encryptionkey@0"
I20260925 10:44:50.787163 14263 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 10:44:50.790496 14278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:44:50.791751 14263 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 10:44:50.791942 14263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/minicluster-data/master-0/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/minicluster-data/master-0/wal
uuid: "68f57a46decf45fda1b4213afe411d80"
format_stamp: "Formatted at 2026-09-25 10:44:50 on dist-test-slave-dxdd"
server_key: "17536d3ab4e142f0427373af01e1c644"
server_key_iv: "f69a515a5bafeb472460df0f432202e0"
server_key_version: "encryptionkey@0"
I20260925 10:44:50.792155 14263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:44:50.815058 14263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:44:50.818857 14263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:44:50.819113 14263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:44:50.837000 14263 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.62:36181
I20260925 10:44:50.837030 14329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.62:36181 every 8 connection(s)
I20260925 10:44:50.838253 14263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/minicluster-data/master-0/data/info.pb
I20260925 10:44:50.841687 14330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:44:50.844247 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 14263
I20260925 10:44:50.844434 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/minicluster-data/master-0/wal/instance
I20260925 10:44:50.844985 23952 external_mini_cluster.cc:1428] Setting key 3d7947109ecb68da685959852bcbec6e
I20260925 10:44:50.850533 14330 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.62" port: 36181 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:44:50 dist-test-slave-dxdd krb5kdc[14250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333090, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.62@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.100.61@KRBTEST.COM; defaulting to no policy
I20260925 10:44:50.872339 14332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:50.854241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:36181 (local address 127.0.0.1:37522)
0925 10:44:50.855099 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:50.855161 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 10:44:50.857059 (+ 1898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:44:50.857870 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:44:50.857877 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:44:50.858588 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:44:50.858937 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:50.858954 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:50.860465 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:50.860474 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:50.861464 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:50.861476 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:50.862396 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:50.862404 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:50.862595 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:50.863259 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:44:50.863279 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:44:50.865421 (+ 2142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:44:50.868446 (+ 3025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:44:50.868458 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:44:50.868468 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:44:50.868837 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:44:50.869219 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:44:50.869223 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:44:50.869227 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:44:50.869371 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:44:50.870050 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:44:50.870058 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:44:50.870254 (+ 196us) client_negotiation.cc:769] Sending connection context
0925 10:44:50.870570 (+ 316us) client_negotiation.cc:240] Negotiation successful
0925 10:44:50.871106 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"mutex_wait_us":138,"thread_start_us":363,"threads_started":1}
I20260925 10:44:50.872390 14333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:50.854241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37522 (local address 127.23.100.62:36181)
0925 10:44:50.855881 (+ 1640us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:50.855890 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:50.855941 (+ 51us) server_negotiation.cc:407] Connection header received
0925 10:44:50.857420 (+ 1479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:50.857426 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:50.857646 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:50.857841 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:50.859147 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:50.860299 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:50.861599 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:50.862155 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:50.862385 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:50.865618 (+ 3233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:50.865625 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:50.865636 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:44:50.865665 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:50.868322 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:50.868968 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:50.868974 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:50.868976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:50.869107 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:50.869477 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:50.869488 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:50.869490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:50.869913 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:50.870105 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:50.870692 (+ 587us) server_negotiation.cc:299] Negotiation successful
0925 10:44:50.870974 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1427,"thread_start_us":1200,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.61@KRBTEST.COM" created.
I20260925 10:44:50.875391 14330 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.61" port: 41659 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:44:50.876608 14332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:50.876104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.61:41659 (local address 127.0.0.1:33780)
0925 10:44:50.876315 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:50.876451 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.61:41659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260925 10:44:50.876886 14280 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.100.61:41659: connect: Connection refused (error 111)
I20260925 10:44:50.878489 14332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:50.878022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.61:41659 (local address 127.0.0.1:33790)
0925 10:44:50.878194 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:50.878281 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.61:41659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260925 10:44:50.878675 14330 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.61:41659: Network error: Client connection negotiation failed: client connection to 127.23.100.61:41659: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.61.keytab.
Entry for principal kudu/127.23.100.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.61.keytab.
I20260925 10:44:50.901155 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.61:41659
--webserver_interface=127.23.100.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.61.keytab
--principal=kudu/127.23.100.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:43715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.62:36181,127.23.100.61:41659,127.23.100.60:36045
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:44:50.928079 14330 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.61" port: 41659 } attempt: 1
I20260925 10:44:50.929270 14332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:50.928799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.61:41659 (local address 127.0.0.1:33806)
0925 10:44:50.928982 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:50.929145 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.61:41659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 10:44:50.930238 14332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:50.929902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.61:41659 (local address 127.0.0.1:33818)
0925 10:44:50.930048 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:50.930132 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.61:41659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 10:44:50.930449 14330 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.61:41659: Network error: Client connection negotiation failed: client connection to 127.23.100.61:41659: connect: Connection refused (error 111)
I20260925 10:44:50.998739 14330 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.61" port: 41659 } attempt: 2
I20260925 10:44:50.999886 14332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:50.999411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.61:41659 (local address 127.0.0.1:33834)
0925 10:44:50.999655 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:50.999740 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.61:41659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260925 10:44:51.000880 14332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:51.000540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.61:41659 (local address 127.0.0.1:33844)
0925 10:44:51.000695 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:51.000758 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.61:41659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 10:44:51.001075 14330 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.61:41659: Network error: Client connection negotiation failed: client connection to 127.23.100.61:41659: connect: Connection refused (error 111)
I20260925 10:44:51.092360 14330 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.61" port: 41659 } attempt: 3
I20260925 10:44:51.093591 14332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:51.093166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.61:41659 (local address 127.0.0.1:33858)
0925 10:44:51.093371 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:51.093461 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.61:41659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260925 10:44:51.094584 14332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:51.094213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.61:41659 (local address 127.0.0.1:33860)
0925 10:44:51.094402 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:51.094467 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.61:41659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 10:44:51.094762 14330 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.61:41659: Network error: Client connection negotiation failed: client connection to 127.23.100.61:41659: connect: Connection refused (error 111)
W20260925 10:44:51.219801 14337 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 10:44:51.220168 14337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:44:51.220292 14337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:44:51.220425 14337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:44:51.227600 14337 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:44:51.227718 14337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:44:51.227835 14337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:44:51.227918 14337 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:44:51.228030 14337 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:44:51.237471 14337 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:43715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.62:36181,127.23.100.61:41659,127.23.100.60:36045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.61.keytab
--principal=kudu/127.23.100.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.61:41659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.100.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:44:51.237984 14330 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.61" port: 41659 } attempt: 4
I20260925 10:44:51.239049 14337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:44:51.239272 14332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:51.238699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.61:41659 (local address 127.0.0.1:33862)
0925 10:44:51.239020 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:51.239103 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.61:41659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182}
I20260925 10:44:51.240260 14332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:51.239897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.61:41659 (local address 127.0.0.1:33868)
0925 10:44:51.240048 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:51.240131 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.61:41659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 10:44:51.240470 14337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:44:51.240470 14330 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.61:41659: Network error: Client connection negotiation failed: client connection to 127.23.100.61:41659: connect: Connection refused (error 111)
W20260925 10:44:51.251665 14345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:51.251717 14342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:51.251740 14343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:44:51.252847 14337 server_base.cc:1034] running on GCE node
I20260925 10:44:51.253593 14337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:44:51.254992 14337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:44:51.256223 14337 hybrid_clock.cc:648] HybridClock initialized: now 1790333091256154 us; error 44 us; skew 500 ppm
I20260925 10:44:51.256522 14337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:44:51 dist-test-slave-dxdd krb5kdc[14250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333091, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:44:51.263458 14337 init.cc:377] Logged in from keytab as kudu/127.23.100.61@KRBTEST.COM (short username kudu)
I20260925 10:44:51.264806 14337 webserver.cc:460] Webserver started at http://127.23.100.61:45387/ using document root <none> and password file <none>
I20260925 10:44:51.265412 14337 fs_manager.cc:359] Metadata directory not provided
I20260925 10:44:51.265496 14337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:44:51.265736 14337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:44:51.268501 14337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/minicluster-data/master-1/data/instance:
uuid: "1a0a601dfc024f9bbc826c816b9fb75f"
format_stamp: "Formatted at 2026-09-25 10:44:51 on dist-test-slave-dxdd"
server_key: "e7bd954477050ed6f4ffc7c860c4f09e"
server_key_iv: "dad636cce8430079cf49697d574fd386"
server_key_version: "encryptionkey@0"
I20260925 10:44:51.269182 14337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/minicluster-data/master-1/wal/instance:
uuid: "1a0a601dfc024f9bbc826c816b9fb75f"
format_stamp: "Formatted at 2026-09-25 10:44:51 on dist-test-slave-dxdd"
server_key: "e7bd954477050ed6f4ffc7c860c4f09e"
server_key_iv: "dad636cce8430079cf49697d574fd386"
server_key_version: "encryptionkey@0"
I20260925 10:44:51.274058 14337 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260925 10:44:51.277233 14352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:44:51.278564 14337 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:44:51.278752 14337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/minicluster-data/master-1/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/minicluster-data/master-1/wal
uuid: "1a0a601dfc024f9bbc826c816b9fb75f"
format_stamp: "Formatted at 2026-09-25 10:44:51 on dist-test-slave-dxdd"
server_key: "e7bd954477050ed6f4ffc7c860c4f09e"
server_key_iv: "dad636cce8430079cf49697d574fd386"
server_key_version: "encryptionkey@0"
I20260925 10:44:51.278970 14337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:44:51.290882 14337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:44:51.294853 14337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:44:51.295125 14337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:44:51.312598 14337 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.61:41659
I20260925 10:44:51.312618 14403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.61:41659 every 8 connection(s)
I20260925 10:44:51.313880 14337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/minicluster-data/master-1/data/info.pb
I20260925 10:44:51.314771 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 14337
I20260925 10:44:51.314982 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/minicluster-data/master-1/wal/instance
I20260925 10:44:51.315376 23952 external_mini_cluster.cc:1428] Setting key cd97bf6e5d2f24fcded5ede24aeedab4
I20260925 10:44:51.317472 14404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:44:51.323812 14404 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.62" port: 36181 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:44:51 dist-test-slave-dxdd krb5kdc[14250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333091, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.61@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.100.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.60@KRBTEST.COM" created.
I20260925 10:44:51.343977 14333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:51.326209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37538 (local address 127.23.100.62:36181)
0925 10:44:51.326465 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:51.326474 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:51.327580 (+ 1106us) server_negotiation.cc:407] Connection header received
0925 10:44:51.329385 (+ 1805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:51.329393 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:51.329497 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:51.329614 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:51.331070 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:51.332177 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:51.333648 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:51.334421 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:51.334564 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:51.338010 (+ 3446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:51.338018 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:51.338025 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:44:51.338073 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:51.340981 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:51.341673 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:51.341681 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:51.341686 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:51.341807 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:51.342228 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:51.342233 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:51.342236 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:51.342572 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:51.342725 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:51.343409 (+ 684us) server_negotiation.cc:299] Negotiation successful
0925 10:44:51.343665 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 10:44:51.344909 14407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:51.326663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:36181 (local address 127.0.0.1:37538)
0925 10:44:51.327410 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:51.327460 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 10:44:51.329117 (+ 1657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:44:51.329790 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:44:51.329797 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:44:51.330287 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:44:51.330879 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:51.330894 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:51.332337 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:51.332340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:51.333498 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:51.333509 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:51.334631 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:51.334636 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:51.334824 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:51.335524 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:44:51.335549 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:44:51.337769 (+ 2220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:44:51.341118 (+ 3349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:44:51.341128 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:44:51.341142 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:44:51.341551 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:44:51.341923 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:44:51.341928 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:44:51.341931 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:44:51.342109 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:44:51.342710 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:44:51.342717 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:44:51.343063 (+ 346us) client_negotiation.cc:769] Sending connection context
0925 10:44:51.343334 (+ 271us) client_negotiation.cc:240] Negotiation successful
0925 10:44:51.343817 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":323,"threads_started":1}
I20260925 10:44:51.346737 14404 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.61" port: 41659 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:44:51 dist-test-slave-dxdd krb5kdc[14250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333091, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.61@KRBTEST.COM for kudu/127.23.100.61@KRBTEST.COM
I20260925 10:44:51.360381 14407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:51.347507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.61:41659 (local address 127.0.0.1:33876)
0925 10:44:51.347712 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:51.347733 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 10:44:51.347837 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:44:51.348753 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:44:51.348757 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:44:51.349183 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:44:51.349305 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:51.349313 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:51.350707 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:51.350713 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:51.351526 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:51.351536 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:51.352321 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:51.352326 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:51.352498 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:51.353047 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:44:51.353063 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:44:51.354978 (+ 1915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:44:51.357918 (+ 2940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:44:51.357925 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:44:51.357930 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:44:51.358403 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:44:51.358755 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:44:51.358761 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:44:51.358782 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:44:51.358917 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:44:51.359466 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:44:51.359476 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:44:51.359602 (+ 126us) client_negotiation.cc:769] Sending connection context
0925 10:44:51.359751 (+ 149us) client_negotiation.cc:240] Negotiation successful
0925 10:44:51.360008 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 10:44:51.360456 14408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:51.347686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33876 (local address 127.23.100.61:41659)
0925 10:44:51.348367 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:51.348373 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:51.348396 (+ 23us) server_negotiation.cc:407] Connection header received
0925 10:44:51.348438 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:51.348445 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:51.348613 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:51.348759 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:51.349482 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:51.350586 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:51.351652 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:51.352168 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:51.352308 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:51.355169 (+ 2861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:51.355179 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:51.355190 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:44:51.355218 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:51.357785 (+ 2567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:51.358526 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:51.358534 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:51.358537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:51.358645 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:51.359023 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:51.359030 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:51.359033 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:51.359338 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:51.359545 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:51.359889 (+ 344us) server_negotiation.cc:299] Negotiation successful
0925 10:44:51.360095 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":401,"threads_started":1}
I20260925 10:44:51.362825 14404 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.60" port: 36045 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:44:51.364061 14407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:51.363661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:36045 (local address 127.0.0.1:49182)
0925 10:44:51.363804 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:51.363901 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.60:36045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 10:44:51.364318 14356 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.100.60:36045: connect: Connection refused (error 111)
I20260925 10:44:51.365711 14407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:51.365394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:36045 (local address 127.0.0.1:49190)
0925 10:44:51.365540 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:51.365621 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.60:36045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260925 10:44:51.365906 14404 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.60:36045: Network error: Client connection negotiation failed: client connection to 127.23.100.60:36045: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.60.keytab.
Entry for principal kudu/127.23.100.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.60.keytab.
I20260925 10:44:51.370005 23952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
/tmp/dist-test-taskVr7Mel/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.60:36045
--webserver_interface=127.23.100.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.60.keytab
--principal=kudu/127.23.100.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.20:43715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.62:36181,127.23.100.61:41659,127.23.100.60:36045
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:44:51.415313 14404 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.60" port: 36045 } attempt: 1
I20260925 10:44:51.416668 14407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:51.416047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:36045 (local address 127.0.0.1:49200)
0925 10:44:51.416363 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:51.416478 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.60:36045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178}
I20260925 10:44:51.417927 14407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:51.417479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:36045 (local address 127.0.0.1:49208)
0925 10:44:51.417666 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:51.417756 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.60:36045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 10:44:51.418154 14404 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.60:36045: Network error: Client connection negotiation failed: client connection to 127.23.100.60:36045: connect: Connection refused (error 111)
I20260925 10:44:51.486459 14404 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.60" port: 36045 } attempt: 2
I20260925 10:44:51.487722 14407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:51.487063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:36045 (local address 127.0.0.1:60566)
0925 10:44:51.487316 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:51.487461 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.60:36045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260925 10:44:51.488937 14407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:51.488453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:36045 (local address 127.0.0.1:60572)
0925 10:44:51.488657 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:51.488762 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.60:36045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 10:44:51.489181 14404 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.60:36045: Network error: Client connection negotiation failed: client connection to 127.23.100.60:36045: connect: Connection refused (error 111)
I20260925 10:44:51.539732 14330 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.61" port: 41659 } attempt: 5
Sep 25 10:44:51 dist-test-slave-dxdd krb5kdc[14250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333090, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.62@KRBTEST.COM for kudu/127.23.100.61@KRBTEST.COM
I20260925 10:44:51.553901 14332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:51.540554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.61:41659 (local address 127.0.0.1:54940)
0925 10:44:51.540779 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:51.540796 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:44:51.540899 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:44:51.541297 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:44:51.541300 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:44:51.541643 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:44:51.541802 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:51.541809 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:51.543240 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:51.543244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:51.544076 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:51.544087 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:51.545072 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:51.545076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:51.545234 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:51.545877 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:44:51.545895 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:44:51.548196 (+ 2301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:44:51.551812 (+ 3616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:44:51.551816 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:44:51.551818 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:44:51.552177 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:44:51.552559 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:44:51.552562 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:44:51.552564 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:44:51.552684 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:44:51.553260 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:44:51.553264 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:44:51.553382 (+ 118us) client_negotiation.cc:769] Sending connection context
0925 10:44:51.553493 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 10:44:51.553672 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260925 10:44:51.554392 14408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:51.540669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54940 (local address 127.23.100.61:41659)
0925 10:44:51.540870 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:51.540876 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:51.540898 (+ 22us) server_negotiation.cc:407] Connection header received
0925 10:44:51.541041 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:51.541046 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:51.541141 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:51.541267 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:51.541975 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:51.543105 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:51.544245 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:51.544929 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:51.545085 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:51.548407 (+ 3322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:51.548413 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:51.548417 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:44:51.548455 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:51.551677 (+ 3222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:51.552290 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:51.552295 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:51.552299 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:51.552455 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:51.552788 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:51.552792 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:51.552796 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:51.553149 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:51.553319 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:51.553646 (+ 327us) server_negotiation.cc:299] Negotiation successful
0925 10:44:51.553906 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 10:44:51.555430 14330 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.60" port: 36045 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:44:51.556644 14332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:51.556114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:36045 (local address 127.0.0.1:60586)
0925 10:44:51.556310 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:51.556444 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.60:36045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 10:44:51.557691 14332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:51.557249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:36045 (local address 127.0.0.1:60592)
0925 10:44:51.557428 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:51.557542 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.60:36045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 10:44:51.557952 14330 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.60:36045: Network error: Client connection negotiation failed: client connection to 127.23.100.60:36045: connect: Connection refused (error 111)
I20260925 10:44:51.580413 14404 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.60" port: 36045 } attempt: 3
I20260925 10:44:51.581633 14407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:51.581042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:36045 (local address 127.0.0.1:60600)
0925 10:44:51.581257 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:51.581392 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.60:36045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260925 10:44:51.582962 14407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:51.582442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:36045 (local address 127.0.0.1:60610)
0925 10:44:51.582730 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:51.582817 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.60:36045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
W20260925 10:44:51.583175 14404 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.60:36045: Network error: Client connection negotiation failed: client connection to 127.23.100.60:36045: connect: Connection refused (error 111)
I20260925 10:44:51.609180 14330 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.60" port: 36045 } attempt: 1
I20260925 10:44:51.610364 14332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:51.609793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:36045 (local address 127.0.0.1:60620)
0925 10:44:51.610025 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:51.610137 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.60:36045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260925 10:44:51.611444 14332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:51.611033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:36045 (local address 127.0.0.1:60626)
0925 10:44:51.611181 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:51.611283 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.60:36045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 10:44:51.611662 14330 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.60:36045: Network error: Client connection negotiation failed: client connection to 127.23.100.60:36045: connect: Connection refused (error 111)
I20260925 10:44:51.679912 14330 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.60" port: 36045 } attempt: 2
I20260925 10:44:51.681129 14332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:51.680603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:36045 (local address 127.0.0.1:60632)
0925 10:44:51.680831 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:51.680938 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.60:36045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260925 10:44:51.682163 14332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:51.681792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:36045 (local address 127.0.0.1:60642)
0925 10:44:51.681934 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:51.682036 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.60:36045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 10:44:51.682413 14330 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.60:36045: Network error: Client connection negotiation failed: client connection to 127.23.100.60:36045: connect: Connection refused (error 111)
W20260925 10:44:51.693347 14411 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 10:44:51.693666 14411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:44:51.693765 14411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:44:51.693922 14411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:44:51.701154 14411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:44:51.701232 14411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:44:51.701354 14411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:44:51.701433 14411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:44:51.701536 14411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:44:51.711040 14411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.20:43715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.62:36181,127.23.100.61:41659,127.23.100.60:36045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/krb5kdc/kudu_127.23.100.60.keytab
--principal=kudu/127.23.100.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.60:36045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.100.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:44:51.712258 14411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:44:51.713531 14411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:44:51.724764 14416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:51.724802 14417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:44:51.724735 14419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:44:51.725598 14411 server_base.cc:1034] running on GCE node
I20260925 10:44:51.726222 14411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:44:51.726393 14404 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.60" port: 36045 } attempt: 4
I20260925 10:44:51.727409 14407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:51.726911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:36045 (local address 127.0.0.1:60658)
0925 10:44:51.727132 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:51.727226 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.60:36045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 10:44:51.727874 14411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:44:51.728502 14407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:51.728183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:36045 (local address 127.0.0.1:60660)
0925 10:44:51.728339 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:51.728404 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.60:36045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 10:44:51.728698 14404 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.60:36045: Network error: Client connection negotiation failed: client connection to 127.23.100.60:36045: connect: Connection refused (error 111)
I20260925 10:44:51.729116 14411 hybrid_clock.cc:648] HybridClock initialized: now 1790333091729060 us; error 26 us; skew 500 ppm
I20260925 10:44:51.729439 14411 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:44:51 dist-test-slave-dxdd krb5kdc[14250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333091, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:44:51.732415 14411 init.cc:377] Logged in from keytab as kudu/127.23.100.60@KRBTEST.COM (short username kudu)
I20260925 10:44:51.733798 14411 webserver.cc:460] Webserver started at http://127.23.100.60:45109/ using document root <none> and password file <none>
I20260925 10:44:51.734460 14411 fs_manager.cc:359] Metadata directory not provided
I20260925 10:44:51.734546 14411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:44:51.734814 14411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:44:51.737537 14411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/minicluster-data/master-2/data/instance:
uuid: "4ac1f9d63d7042a7a9936a6765af8fc0"
format_stamp: "Formatted at 2026-09-25 10:44:51 on dist-test-slave-dxdd"
server_key: "2f1bbe8d31647be7e94f7c2739c9b9c6"
server_key_iv: "32fc183fed6ad4e80f581f99fcbce09e"
server_key_version: "encryptionkey@0"
I20260925 10:44:51.738255 14411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/minicluster-data/master-2/wal/instance:
uuid: "4ac1f9d63d7042a7a9936a6765af8fc0"
format_stamp: "Formatted at 2026-09-25 10:44:51 on dist-test-slave-dxdd"
server_key: "2f1bbe8d31647be7e94f7c2739c9b9c6"
server_key_iv: "32fc183fed6ad4e80f581f99fcbce09e"
server_key_version: "encryptionkey@0"
I20260925 10:44:51.743155 14411 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 10:44:51.746219 14426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:44:51.747408 14411 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 10:44:51.747572 14411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/minicluster-data/master-2/data,/tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/minicluster-data/master-2/wal
uuid: "4ac1f9d63d7042a7a9936a6765af8fc0"
format_stamp: "Formatted at 2026-09-25 10:44:51 on dist-test-slave-dxdd"
server_key: "2f1bbe8d31647be7e94f7c2739c9b9c6"
server_key_iv: "32fc183fed6ad4e80f581f99fcbce09e"
server_key_version: "encryptionkey@0"
I20260925 10:44:51.747788 14411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:44:51.760425 14411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:44:51.764338 14411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:44:51.764612 14411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:44:51.782603 14411 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.60:36045
I20260925 10:44:51.782610 14477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.60:36045 every 8 connection(s)
I20260925 10:44:51.783998 14411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/minicluster-data/master-2/data/info.pb
I20260925 10:44:51.787590 14478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:44:51.788633 14330 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.60" port: 36045 } attempt: 3
I20260925 10:44:51.794327 23952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu as pid 14411
I20260925 10:44:51.794507 23952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVr7Mel/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790332740062128-23952-0/minicluster-data/master-2/wal/instance
I20260925 10:44:51.795015 23952 external_mini_cluster.cc:1428] Setting key 053194a71b4e51cdc365560d13e393ec
I20260925 10:44:51.796397 14478 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.62" port: 36181 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:44:51 dist-test-slave-dxdd krb5kdc[14250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333090, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.62@KRBTEST.COM for kudu/127.23.100.60@KRBTEST.COM
I20260925 10:44:51.814491 14332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:51.789211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:36045 (local address 127.0.0.1:60668)
0925 10:44:51.789430 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:51.789453 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 10:44:51.789553 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:44:51.793969 (+ 4416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:44:51.793975 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:44:51.794503 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:44:51.794705 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:51.794715 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:51.797372 (+ 2657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:51.797382 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:51.799389 (+ 2007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:51.799411 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:51.801319 (+ 1908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:51.801328 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:51.801590 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:51.802424 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:44:51.802456 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:44:51.806017 (+ 3561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:44:51.811237 (+ 5220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:44:51.811242 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:44:51.811245 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:44:51.811787 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:44:51.812245 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:44:51.812253 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:44:51.812256 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:44:51.812399 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:44:51.813386 (+ 987us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:44:51.813391 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:44:51.813603 (+ 212us) client_negotiation.cc:769] Sending connection context
0925 10:44:51.813754 (+ 151us) client_negotiation.cc:240] Negotiation successful
0925 10:44:51.814037 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 10:44:51.816026 14480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:51.790152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60668 (local address 127.23.100.60:36045)
0925 10:44:51.790911 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:51.790922 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:51.790958 (+ 36us) server_negotiation.cc:407] Connection header received
0925 10:44:51.793203 (+ 2245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:51.793222 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:51.793611 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:51.794001 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:51.794871 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:51.797117 (+ 2246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:51.799708 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:51.801100 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:51.802070 (+ 970us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:51.806454 (+ 4384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:51.806468 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:51.806491 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:44:51.806548 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:51.811076 (+ 4528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:51.811918 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:51.811926 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:51.811934 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:51.812124 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:51.812521 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:51.812528 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:51.812533 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:51.813238 (+ 705us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:51.813432 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:51.814074 (+ 642us) server_negotiation.cc:299] Negotiation successful
0925 10:44:51.814600 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":362,"threads_started":1}
Sep 25 10:44:51 dist-test-slave-dxdd krb5kdc[14250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333091, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.60@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
Sep 25 10:44:51 dist-test-slave-dxdd krb5kdc[14250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.60@KRBTEST.COM
I20260925 10:44:51.823300 14330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68f57a46decf45fda1b4213afe411d80: Bootstrap starting.
I20260925 10:44:51.825997 14333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:51.800689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46054 (local address 127.23.100.62:36181)
0925 10:44:51.800905 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:51.800913 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:51.809621 (+ 8708us) server_negotiation.cc:407] Connection header received
0925 10:44:51.809796 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:51.809801 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:51.809876 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:51.809998 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:51.811665 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:51.812689 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:51.814546 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:51.815242 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:51.815309 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:51.819228 (+ 3919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:51.819235 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:51.819239 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:44:51.819273 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:51.822863 (+ 3590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:51.823871 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:51.823876 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:51.823879 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:51.824037 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:51.824404 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:51.824408 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:51.824410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:51.824686 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:51.824887 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:51.825444 (+ 557us) server_negotiation.cc:299] Negotiation successful
0925 10:44:51.825635 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 10:44:51.826068 14483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:51.800120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.62:36181 (local address 127.0.0.1:46054)
0925 10:44:51.809464 (+ 9344us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:51.809510 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 10:44:51.809677 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:44:51.810071 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:44:51.810078 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:44:51.811088 (+ 1010us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:44:51.811506 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:51.811519 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:51.812851 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:51.812856 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:51.814335 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:51.814360 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:51.815550 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:51.815554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:51.815743 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:51.816354 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:44:51.816374 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:44:51.818997 (+ 2623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:44:51.823008 (+ 4011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:44:51.823020 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:44:51.823028 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:44:51.823729 (+ 701us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:44:51.824140 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:44:51.824143 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:44:51.824145 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:44:51.824277 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:44:51.824974 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:44:51.824981 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:44:51.825149 (+ 168us) client_negotiation.cc:769] Sending connection context
0925 10:44:51.825320 (+ 171us) client_negotiation.cc:240] Negotiation successful
0925 10:44:51.825634 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9063,"thread_start_us":8238,"threads_started":1}
I20260925 10:44:51.827245 14330 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68f57a46decf45fda1b4213afe411d80: Neither blocks nor log segments found. Creating new log.
I20260925 10:44:51.827394 14478 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.61" port: 41659 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:44:51.827844 14482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:51.795888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60670 (local address 127.23.100.60:36045)
0925 10:44:51.807334 (+ 11446us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:51.807340 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:51.808751 (+ 1411us) server_negotiation.cc:407] Connection header received
0925 10:44:51.808929 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:51.808934 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:51.809015 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:51.809122 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:51.811181 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:51.812455 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:51.814095 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:51.814789 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:51.814887 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:51.820693 (+ 5806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:51.820698 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:51.820701 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:44:51.820736 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:51.823739 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:51.824821 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:51.824826 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:51.824829 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:51.824974 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:51.825434 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:51.825439 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:51.825443 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:51.825782 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:51.826037 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:51.827252 (+ 1215us) server_negotiation.cc:299] Negotiation successful
0925 10:44:51.827455 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11267,"thread_start_us":11995,"threads_started":1}
I20260925 10:44:51.828511 14330 log.cc:824] T 00000000000000000000000000000000 P 68f57a46decf45fda1b4213afe411d80: Log is configured to *not* fsync() on all Append() calls
Sep 25 10:44:51 dist-test-slave-dxdd krb5kdc[14250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333091, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.60@KRBTEST.COM for kudu/127.23.100.61@KRBTEST.COM
I20260925 10:44:51.837852 14330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68f57a46decf45fda1b4213afe411d80: No bootstrap required, opened a new log
Sep 25 10:44:51 dist-test-slave-dxdd krb5kdc[14250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.62@KRBTEST.COM
I20260925 10:44:51.842979 14408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:51.828383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54956 (local address 127.23.100.61:41659)
0925 10:44:51.828609 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:51.828616 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:51.828875 (+ 259us) server_negotiation.cc:407] Connection header received
0925 10:44:51.829061 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:51.829069 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:51.829162 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:51.829280 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:51.830347 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:51.831281 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:51.832412 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:51.832860 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:51.832987 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:51.836531 (+ 3544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:51.836538 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:51.836543 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:44:51.836582 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:51.839975 (+ 3393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:51.840757 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:51.840765 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:51.840769 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:51.840920 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:51.841356 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:51.841361 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:51.841364 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:51.841670 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:51.841796 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:51.842381 (+ 585us) server_negotiation.cc:299] Negotiation successful
0925 10:44:51.842607 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 10:44:51.843708 14483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:51.828378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.61:41659 (local address 127.0.0.1:54956)
0925 10:44:51.828766 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:51.828785 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:44:51.828894 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:44:51.829330 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:44:51.829338 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:44:51.829953 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:44:51.830174 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:51.830188 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:51.831415 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:51.831421 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:51.832266 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:51.832276 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:51.833039 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:51.833049 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:51.833321 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:51.833940 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:44:51.833969 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:44:51.836309 (+ 2340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:44:51.840142 (+ 3833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:44:51.840149 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:44:51.840154 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:44:51.840620 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:44:51.841060 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:44:51.841065 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:44:51.841069 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:44:51.841227 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:44:51.841893 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:44:51.841898 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:44:51.842067 (+ 169us) client_negotiation.cc:769] Sending connection context
0925 10:44:51.843231 (+ 1164us) client_negotiation.cc:240] Negotiation successful
0925 10:44:51.843503 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253}
I20260925 10:44:51.844745 14478 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.60" port: 36045 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:44:51.849016 14333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:51.830143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46064 (local address 127.23.100.62:36181)
0925 10:44:51.830290 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:51.830296 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:51.830624 (+ 328us) server_negotiation.cc:407] Connection header received
0925 10:44:51.830818 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:51.830823 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:51.830920 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:51.831009 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:51.832851 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:51.833815 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:51.835346 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:51.835877 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:51.836017 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:51.841435 (+ 5418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:51.841444 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:51.841449 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:44:51.841480 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:51.845014 (+ 3534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:51.845813 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:51.845818 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:51.845821 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:51.845996 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:51.846453 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:51.846457 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:51.846460 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:51.847124 (+ 664us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:51.847475 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:51.848514 (+ 1039us) server_negotiation.cc:299] Negotiation successful
0925 10:44:51.848713 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 10:44:51.850786 14330 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68f57a46decf45fda1b4213afe411d80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f57a46decf45fda1b4213afe411d80" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 36181 } } peers { permanent_uuid: "1a0a601dfc024f9bbc826c816b9fb75f" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 41659 } } peers { permanent_uuid: "4ac1f9d63d7042a7a9936a6765af8fc0" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 36045 } }
I20260925 10:44:51.851176 14330 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68f57a46decf45fda1b4213afe411d80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68f57a46decf45fda1b4213afe411d80, State: Initialized, Role: FOLLOWER
I20260925 10:44:51.851672 14330 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68f57a46decf45fda1b4213afe411d80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f57a46decf45fda1b4213afe411d80" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 36181 } } peers { permanent_uuid: "1a0a601dfc024f9bbc826c816b9fb75f" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 41659 } } peers { permanent_uuid: "4ac1f9d63d7042a7a9936a6765af8fc0" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 36045 } }
Sep 25 10:44:51 dist-test-slave-dxdd krb5kdc[14250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333091, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.60@KRBTEST.COM for kudu/127.23.100.60@KRBTEST.COM
I20260925 10:44:51.854602 14485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68f57a46decf45fda1b4213afe411d80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f57a46decf45fda1b4213afe411d80" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 36181 } } peers { permanent_uuid: "1a0a601dfc024f9bbc826c816b9fb75f" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 41659 } } peers { permanent_uuid: "4ac1f9d63d7042a7a9936a6765af8fc0" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 36045 } } }
I20260925 10:44:51.854985 14485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68f57a46decf45fda1b4213afe411d80 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:44:51.855909 14330 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68f57a46decf45fda1b4213afe411d80 [sys.catalog]: configured and running, proceeding with master startup.
Sep 25 10:44:51 dist-test-slave-dxdd krb5kdc[14250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.61@KRBTEST.COM
I20260925 10:44:51.860931 14482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:51.845685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60684 (local address 127.23.100.60:36045)
0925 10:44:51.845865 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:51.845869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:51.846543 (+ 674us) server_negotiation.cc:407] Connection header received
0925 10:44:51.846724 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:51.846729 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:51.846790 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:51.846859 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:51.847793 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:51.848811 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:51.850423 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:51.851006 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:51.851119 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:51.854555 (+ 3436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:51.854560 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:51.854564 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:44:51.854598 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:51.857467 (+ 2869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:51.858363 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:51.858372 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:51.858377 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:51.858544 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:51.859039 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:51.859047 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:51.859053 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:51.859479 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:51.859636 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:51.860211 (+ 575us) server_negotiation.cc:299] Negotiation successful
0925 10:44:51.860476 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 10:44:51.861967 14483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:51.845401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:36045 (local address 127.0.0.1:60684)
0925 10:44:51.846474 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:51.846491 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:44:51.846613 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:44:51.847019 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:44:51.847023 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:44:51.847478 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:44:51.847658 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:51.847668 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:51.848973 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:51.848978 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:51.850265 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:51.850279 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:51.851308 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:51.851316 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:51.851498 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:51.852060 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:44:51.852080 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:44:51.854403 (+ 2323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:44:51.857588 (+ 3185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:44:51.857592 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:44:51.857595 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:44:51.858167 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:44:51.858681 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:44:51.858688 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:44:51.858694 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:44:51.858912 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:44:51.859712 (+ 800us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:44:51.859719 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:44:51.859924 (+ 205us) client_negotiation.cc:769] Sending connection context
0925 10:44:51.861109 (+ 1185us) client_negotiation.cc:240] Negotiation successful
0925 10:44:51.861470 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":960}
I20260925 10:44:51.867533 14408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:51.851059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54962 (local address 127.23.100.61:41659)
0925 10:44:51.851237 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:51.851242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:51.851500 (+ 258us) server_negotiation.cc:407] Connection header received
0925 10:44:51.851630 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:51.851636 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:51.851730 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:51.851836 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:51.854072 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:51.855126 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:51.856741 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:51.857456 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:51.857586 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:51.861850 (+ 4264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:51.861856 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:51.861862 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:44:51.861895 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:51.864749 (+ 2854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:51.865352 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:51.865358 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:51.865366 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:51.865484 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:51.865919 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:51.865924 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:51.865928 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:51.866328 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:51.866586 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:51.866996 (+ 410us) server_negotiation.cc:299] Negotiation successful
0925 10:44:51.867205 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 10:44:51.868363 14478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ac1f9d63d7042a7a9936a6765af8fc0: Bootstrap starting.
I20260925 10:44:51.872021 23952 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:44:51.872742 14478 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ac1f9d63d7042a7a9936a6765af8fc0: Neither blocks nor log segments found. Creating new log.
I20260925 10:44:51.873723 14478 log.cc:824] T 00000000000000000000000000000000 P 4ac1f9d63d7042a7a9936a6765af8fc0: Log is configured to *not* fsync() on all Append() calls
W20260925 10:44:51.876847 14496 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 68f57a46decf45fda1b4213afe411d80: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:44:51.876979 14496 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:44:51.877135 14478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ac1f9d63d7042a7a9936a6765af8fc0: No bootstrap required, opened a new log
I20260925 10:44:51.877045 14305 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "68f57a46decf45fda1b4213afe411d80"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46064
I20260925 10:44:51.877430 14305 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68f57a46decf45fda1b4213afe411d80 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 10:44:51.877629 14305 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68f57a46decf45fda1b4213afe411d80 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:44:51.881176 14305 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68f57a46decf45fda1b4213afe411d80 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f57a46decf45fda1b4213afe411d80" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 36181 } } peers { permanent_uuid: "1a0a601dfc024f9bbc826c816b9fb75f" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 41659 } } peers { permanent_uuid: "4ac1f9d63d7042a7a9936a6765af8fc0" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 36045 } }
I20260925 10:44:51.883507 14305 leader_election.cc:288] T 00000000000000000000000000000000 P 68f57a46decf45fda1b4213afe411d80 [CANDIDATE]: Term 1 election: Requested vote from peers 1a0a601dfc024f9bbc826c816b9fb75f (127.23.100.61:41659), 4ac1f9d63d7042a7a9936a6765af8fc0 (127.23.100.60:36045)
I20260925 10:44:51.888302 14478 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ac1f9d63d7042a7a9936a6765af8fc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f57a46decf45fda1b4213afe411d80" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 36181 } } peers { permanent_uuid: "1a0a601dfc024f9bbc826c816b9fb75f" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 41659 } } peers { permanent_uuid: "4ac1f9d63d7042a7a9936a6765af8fc0" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 36045 } }
I20260925 10:44:51.888787 14478 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ac1f9d63d7042a7a9936a6765af8fc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ac1f9d63d7042a7a9936a6765af8fc0, State: Initialized, Role: FOLLOWER
I20260925 10:44:51.889357 14478 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ac1f9d63d7042a7a9936a6765af8fc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f57a46decf45fda1b4213afe411d80" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 36181 } } peers { permanent_uuid: "1a0a601dfc024f9bbc826c816b9fb75f" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 41659 } } peers { permanent_uuid: "4ac1f9d63d7042a7a9936a6765af8fc0" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 36045 } }
I20260925 10:44:51.883337 14263 master_runner.cc:186] Error getting master registration for 127.23.100.61:41659: OK, instance_id { permanent_uuid: "1a0a601dfc024f9bbc826c816b9fb75f" instance_seqno: 1790333091307744 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 10:44:51.905722 14263 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 10:44:51.906879 14500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ac1f9d63d7042a7a9936a6765af8fc0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f57a46decf45fda1b4213afe411d80" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 36181 } } peers { permanent_uuid: "1a0a601dfc024f9bbc826c816b9fb75f" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 41659 } } peers { permanent_uuid: "4ac1f9d63d7042a7a9936a6765af8fc0" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 36045 } } }
I20260925 10:44:51.907830 14500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ac1f9d63d7042a7a9936a6765af8fc0 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:44:51.908210 14478 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ac1f9d63d7042a7a9936a6765af8fc0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:44:51.908236 14453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "68f57a46decf45fda1b4213afe411d80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4ac1f9d63d7042a7a9936a6765af8fc0"
I20260925 10:44:51.908771 14453 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ac1f9d63d7042a7a9936a6765af8fc0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:44:51.912163 14453 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4ac1f9d63d7042a7a9936a6765af8fc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68f57a46decf45fda1b4213afe411d80 in term 1.
I20260925 10:44:51.912864 14282 leader_election.cc:302] T 00000000000000000000000000000000 P 68f57a46decf45fda1b4213afe411d80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ac1f9d63d7042a7a9936a6765af8fc0, 68f57a46decf45fda1b4213afe411d80; no voters:
I20260925 10:44:51.913425 14485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68f57a46decf45fda1b4213afe411d80 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:44:51.924630 14485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68f57a46decf45fda1b4213afe411d80 [term 1 LEADER]: Becoming Leader. State: Replica: 68f57a46decf45fda1b4213afe411d80, State: Running, Role: LEADER
I20260925 10:44:51.925649 14485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68f57a46decf45fda1b4213afe411d80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f57a46decf45fda1b4213afe411d80" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 36181 } } peers { permanent_uuid: "1a0a601dfc024f9bbc826c816b9fb75f" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 41659 } } peers { permanent_uuid: "4ac1f9d63d7042a7a9936a6765af8fc0" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 36045 } }
I20260925 10:44:51.934262 14504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68f57a46decf45fda1b4213afe411d80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68f57a46decf45fda1b4213afe411d80. Latest consensus state: current_term: 1 leader_uuid: "68f57a46decf45fda1b4213afe411d80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f57a46decf45fda1b4213afe411d80" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 36181 } } peers { permanent_uuid: "1a0a601dfc024f9bbc826c816b9fb75f" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 41659 } } peers { permanent_uuid: "4ac1f9d63d7042a7a9936a6765af8fc0" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 36045 } } }
I20260925 10:44:51.935019 14504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68f57a46decf45fda1b4213afe411d80 [sys.catalog]: This master's current role is: LEADER
I20260925 10:44:51.940486 14507 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:44:51.949378 14507 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:44:51.983215 14333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:51.939991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46068 (local address 127.23.100.62:36181)
0925 10:44:51.940340 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:51.940358 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:51.947830 (+ 7472us) server_negotiation.cc:407] Connection header received
0925 10:44:51.948781 (+ 951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:51.948787 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:51.948876 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:51.948948 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:51.959003 (+ 10055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:51.960128 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:51.966401 (+ 6273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:51.967024 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:51.967123 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:51.973332 (+ 6209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:51.973339 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:51.973344 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:44:51.973395 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:51.977137 (+ 3742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:51.979468 (+ 2331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:51.979477 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:51.979483 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:51.979699 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:51.980230 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:51.980245 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:51.980248 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:51.980723 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:51.981038 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:51.981519 (+ 481us) server_negotiation.cc:299] Negotiation successful
0925 10:44:51.982123 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233}
I20260925 10:44:51.987552 14453 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4ac1f9d63d7042a7a9936a6765af8fc0 [term 1 FOLLOWER]: Refusing update from remote peer 68f57a46decf45fda1b4213afe411d80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:44:51.988615 14504 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 68f57a46decf45fda1b4213afe411d80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ac1f9d63d7042a7a9936a6765af8fc0" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 36045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:44:51.997820 14408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:51.940162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54966 (local address 127.23.100.61:41659)
0925 10:44:51.940511 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:51.940517 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:51.958483 (+ 17966us) server_negotiation.cc:407] Connection header received
0925 10:44:51.958915 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:51.958923 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:51.959010 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:51.959084 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:51.964495 (+ 5411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:51.965577 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:51.967474 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:51.968090 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:51.968194 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:51.985014 (+ 16820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:51.985021 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:51.985025 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:44:51.985065 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:51.989824 (+ 4759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:51.990761 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:51.990769 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:51.990773 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:51.990952 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:51.991442 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:51.991450 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:51.991456 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:51.991808 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:51.992031 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:51.997229 (+ 5198us) server_negotiation.cc:299] Negotiation successful
0925 10:44:51.997465 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219}
I20260925 10:44:51.999660 14521 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 10:44:52.003540 14500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ac1f9d63d7042a7a9936a6765af8fc0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68f57a46decf45fda1b4213afe411d80. Latest consensus state: current_term: 1 leader_uuid: "68f57a46decf45fda1b4213afe411d80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f57a46decf45fda1b4213afe411d80" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 36181 } } peers { permanent_uuid: "1a0a601dfc024f9bbc826c816b9fb75f" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 41659 } } peers { permanent_uuid: "4ac1f9d63d7042a7a9936a6765af8fc0" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 36045 } } }
I20260925 10:44:52.003830 14500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ac1f9d63d7042a7a9936a6765af8fc0 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:44:52.006891 14411 master_runner.cc:186] Error getting master registration for 127.23.100.61:41659: OK, instance_id { permanent_uuid: "1a0a601dfc024f9bbc826c816b9fb75f" instance_seqno: 1790333091307744 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 10:44:52.017807 14411 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 10:44:52.020269 14482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:51.939070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60696 (local address 127.23.100.60:36045)
0925 10:44:51.939284 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:51.939292 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:51.979401 (+ 40109us) server_negotiation.cc:407] Connection header received
0925 10:44:51.981594 (+ 2193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:51.981602 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:51.981712 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:51.981800 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:51.986926 (+ 5126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:51.988105 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:51.992166 (+ 4061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:51.992901 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:51.993023 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:51.996215 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:51.996223 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:51.996227 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:44:51.996263 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:52.001472 (+ 5209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:52.005979 (+ 4507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:52.005985 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:52.005989 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:52.006134 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:52.018286 (+ 12152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:52.018291 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:52.018295 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:52.018699 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:52.019022 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:52.019632 (+ 610us) server_negotiation.cc:299] Negotiation successful
0925 10:44:52.019880 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 10:44:52.022681 14485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68f57a46decf45fda1b4213afe411d80 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "68f57a46decf45fda1b4213afe411d80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f57a46decf45fda1b4213afe411d80" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 36181 } } peers { permanent_uuid: "1a0a601dfc024f9bbc826c816b9fb75f" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 41659 } } peers { permanent_uuid: "4ac1f9d63d7042a7a9936a6765af8fc0" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 36045 } } }
I20260925 10:44:52.023146 14485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68f57a46decf45fda1b4213afe411d80 [sys.catalog]: This master's current role is: LEADER
I20260925 10:44:52.023797 14507 catalog_manager.cc:1347] Generated new cluster ID: 7595e744723845a29469ce351e67113b
I20260925 10:44:52.023890 14507 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:44:52.024066 14500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ac1f9d63d7042a7a9936a6765af8fc0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "68f57a46decf45fda1b4213afe411d80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f57a46decf45fda1b4213afe411d80" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 36181 } } peers { permanent_uuid: "1a0a601dfc024f9bbc826c816b9fb75f" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 41659 } } peers { permanent_uuid: "4ac1f9d63d7042a7a9936a6765af8fc0" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 36045 } } }
I20260925 10:44:52.024264 14500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ac1f9d63d7042a7a9936a6765af8fc0 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:44:52.027932 14404 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.60" port: 36045 } attempt: 5
Sep 25 10:44:52 dist-test-slave-dxdd krb5kdc[14250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790333091, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.61@KRBTEST.COM for kudu/127.23.100.60@KRBTEST.COM
I20260925 10:44:52.046700 14407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:52.028493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.60:36045 (local address 127.0.0.1:60712)
0925 10:44:52.028743 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0925 10:44:52.028765 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 10:44:52.028905 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:44:52.030096 (+ 1191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:44:52.030101 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:44:52.030658 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:44:52.030872 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:52.030884 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:52.031927 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:52.031933 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:52.033433 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:44:52.033448 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:44:52.035812 (+ 2364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:44:52.035818 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:44:52.036050 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:52.036691 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:44:52.036712 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:44:52.039530 (+ 2818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:44:52.043205 (+ 3675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:44:52.043210 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:44:52.043239 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:44:52.043977 (+ 738us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:44:52.044631 (+ 654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:44:52.044636 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:44:52.044658 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:44:52.044875 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:44:52.045686 (+ 811us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:44:52.045691 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:44:52.045833 (+ 142us) client_negotiation.cc:769] Sending connection context
0925 10:44:52.045993 (+ 160us) client_negotiation.cc:240] Negotiation successful
0925 10:44:52.046266 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130}
I20260925 10:44:52.047092 14482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:52.028556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60712 (local address 127.23.100.60:36045)
0925 10:44:52.028701 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:52.028709 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:52.029614 (+ 905us) server_negotiation.cc:407] Connection header received
0925 10:44:52.029669 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:52.029674 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:52.029744 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:52.029828 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:52.031014 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:52.031793 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:52.034953 (+ 3160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:52.035675 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:52.035793 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:52.039701 (+ 3908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:44:52.039707 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:44:52.039713 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:44:52.039747 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:44:52.043037 (+ 3290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:52.044165 (+ 1128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:52.044171 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:52.044175 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:52.044416 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:44:52.045001 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:44:52.045006 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:44:52.045009 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:44:52.045485 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:44:52.045627 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:52.046169 (+ 542us) server_negotiation.cc:299] Negotiation successful
0925 10:44:52.046572 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 10:44:52.052549 14404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a0a601dfc024f9bbc826c816b9fb75f: Bootstrap starting.
I20260925 10:44:52.056566 14404 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a0a601dfc024f9bbc826c816b9fb75f: Neither blocks nor log segments found. Creating new log.
I20260925 10:44:52.057776 14404 log.cc:824] T 00000000000000000000000000000000 P 1a0a601dfc024f9bbc826c816b9fb75f: Log is configured to *not* fsync() on all Append() calls
I20260925 10:44:52.067029 14404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a0a601dfc024f9bbc826c816b9fb75f: No bootstrap required, opened a new log
I20260925 10:44:52.074949 14507 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:44:52.076713 14507 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:44:52.078282 14404 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a0a601dfc024f9bbc826c816b9fb75f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f57a46decf45fda1b4213afe411d80" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 36181 } } peers { permanent_uuid: "1a0a601dfc024f9bbc826c816b9fb75f" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 41659 } } peers { permanent_uuid: "4ac1f9d63d7042a7a9936a6765af8fc0" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 36045 } }
I20260925 10:44:52.078679 14404 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a0a601dfc024f9bbc826c816b9fb75f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a0a601dfc024f9bbc826c816b9fb75f, State: Initialized, Role: FOLLOWER
I20260925 10:44:52.079149 14404 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a0a601dfc024f9bbc826c816b9fb75f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f57a46decf45fda1b4213afe411d80" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 36181 } } peers { permanent_uuid: "1a0a601dfc024f9bbc826c816b9fb75f" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 41659 } } peers { permanent_uuid: "4ac1f9d63d7042a7a9936a6765af8fc0" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 36045 } }
I20260925 10:44:52.087148 14527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a0a601dfc024f9bbc826c816b9fb75f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f57a46decf45fda1b4213afe411d80" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 36181 } } peers { permanent_uuid: "1a0a601dfc024f9bbc826c816b9fb75f" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 41659 } } peers { permanent_uuid: "4ac1f9d63d7042a7a9936a6765af8fc0" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 36045 } } }
I20260925 10:44:52.087846 14527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a0a601dfc024f9bbc826c816b9fb75f [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:44:52.088088 14404 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a0a601dfc024f9bbc826c816b9fb75f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:44:52.088167 14379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "68f57a46decf45fda1b4213afe411d80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1a0a601dfc024f9bbc826c816b9fb75f"
I20260925 10:44:52.088599 14379 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a0a601dfc024f9bbc826c816b9fb75f [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:44:52.095721 14507 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68f57a46decf45fda1b4213afe411d80: Generated new TSK 0
I20260925 10:44:52.097147 14507 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:44:52.097700 14379 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1a0a601dfc024f9bbc826c816b9fb75f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68f57a46decf45fda1b4213afe411d80 in term 1.
I20260925 10:44:52.098400 14378 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1a0a601dfc024f9bbc826c816b9fb75f [term 1 FOLLOWER]: Refusing update from remote peer 68f57a46decf45fda1b4213afe411d80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:44:52.100091 14485 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 68f57a46decf45fda1b4213afe411d80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a0a601dfc024f9bbc826c816b9fb75f" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 41659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:44:52.159508 14536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a0a601dfc024f9bbc826c816b9fb75f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "68f57a46decf45fda1b4213afe411d80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f57a46decf45fda1b4213afe411d80" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 36181 } } peers { permanent_uuid: "1a0a601dfc024f9bbc826c816b9fb75f" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 41659 } } peers { permanent_uuid: "4ac1f9d63d7042a7a9936a6765af8fc0" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 36045 } } }
I20260925 10:44:52.159964 14536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a0a601dfc024f9bbc826c816b9fb75f [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:44:52.160686 14527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a0a601dfc024f9bbc826c816b9fb75f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68f57a46decf45fda1b4213afe411d80. Latest consensus state: current_term: 1 leader_uuid: "68f57a46decf45fda1b4213afe411d80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f57a46decf45fda1b4213afe411d80" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 36181 } } peers { permanent_uuid: "1a0a601dfc024f9bbc826c816b9fb75f" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 41659 } } peers { permanent_uuid: "4ac1f9d63d7042a7a9936a6765af8fc0" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 36045 } } }
I20260925 10:44:52.160957 14527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a0a601dfc024f9bbc826c816b9fb75f [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:44:52.173723 14545 catalog_manager.cc:1259] Loaded cluster ID: 7595e744723845a29469ce351e67113b
I20260925 10:44:52.173914 14545 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1a0a601dfc024f9bbc826c816b9fb75f: loading cluster ID for follower catalog manager: success
I20260925 10:44:52.177670 14545 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1a0a601dfc024f9bbc826c816b9fb75f: acquiring CA information for follower catalog manager: success
I20260925 10:44:52.180128 14545 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1a0a601dfc024f9bbc826c816b9fb75f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 10:44:52.201037 14333 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:44:52.188807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46082 (local address 127.23.100.62:36181)
0925 10:44:52.188985 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:52.188991 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:52.198580 (+ 9589us) server_negotiation.cc:407] Connection header received
0925 10:44:52.199487 (+ 907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:52.199494 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:52.199598 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:52.199669 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:52.200786 (+ 1117us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46082: BlockingRecv error: recv got EOF from 127.0.0.1:46082 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260925 10:44:52.204427 14408 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:44:52.189111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54982 (local address 127.23.100.61:41659)
0925 10:44:52.189277 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:52.189282 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:52.202340 (+ 13058us) server_negotiation.cc:407] Connection header received
0925 10:44:52.203221 (+ 881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:52.203229 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:52.203317 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:52.203393 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:52.204207 (+ 814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54982: BlockingRecv error: recv got EOF from 127.0.0.1:54982 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260925 10:44:52.204423 14482 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:44:52.194604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60714 (local address 127.23.100.60:36045)
0925 10:44:52.194775 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:52.194782 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:52.202916 (+ 8134us) server_negotiation.cc:407] Connection header received
0925 10:44:52.203090 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:52.203098 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:52.203197 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:52.203258 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:52.204211 (+ 953us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60714: BlockingRecv error: recv got EOF from 127.0.0.1:60714 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 10:44:52.215647 14408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:52.209971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54986 (local address 127.23.100.61:41659)
0925 10:44:52.210120 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:52.210128 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:52.210144 (+ 16us) server_negotiation.cc:407] Connection header received
0925 10:44:52.210405 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:52.210415 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:52.210541 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:52.210696 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:44:52.211124 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:52.211843 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:52.213143 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:52.213664 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:52.213792 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:52.214295 (+ 503us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:44:52.214871 (+ 576us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:44:52.215093 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:52.215196 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 10:44:52.215267 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260925 10:44:52.223465 14482 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:44:52.221360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60730 (local address 127.23.100.60:36045)
0925 10:44:52.221583 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:52.221592 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:52.222074 (+ 482us) server_negotiation.cc:407] Connection header received
0925 10:44:52.222293 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:52.222297 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:52.222463 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:52.222571 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:52.223272 (+ 701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60730: BlockingRecv error: recv got EOF from 127.0.0.1:60730 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 10:44:52.232605 14408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:52.227419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55000 (local address 127.23.100.61:41659)
0925 10:44:52.227617 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:52.227625 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:52.228097 (+ 472us) server_negotiation.cc:407] Connection header received
0925 10:44:52.228328 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:52.228333 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:52.228417 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:52.228537 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:44:52.228923 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:52.229613 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:52.230936 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:52.231357 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:52.231485 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:52.231829 (+ 344us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:44:52.231957 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:44:52.232070 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:52.232224 (+ 154us) server_negotiation.cc:299] Negotiation successful
0925 10:44:52.232295 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260925 10:44:52.244132 14482 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:44:52.238612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60740 (local address 127.23.100.60:36045)
0925 10:44:52.238851 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:52.238859 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:52.242819 (+ 3960us) server_negotiation.cc:407] Connection header received
0925 10:44:52.243010 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:52.243016 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:52.243094 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:52.243206 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:52.243896 (+ 690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60740: BlockingRecv error: recv got EOF from 127.0.0.1:60740 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 10:44:52.254379 14408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:52.248628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55014 (local address 127.23.100.61:41659)
0925 10:44:52.248823 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:52.248831 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:52.249301 (+ 470us) server_negotiation.cc:407] Connection header received
0925 10:44:52.249490 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:52.249495 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:52.249566 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:52.249658 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:44:52.250109 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:52.251049 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:52.252544 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:52.253001 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:52.253140 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:52.253515 (+ 375us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:44:52.253676 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:44:52.253771 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:52.253915 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 10:44:52.253999 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260925 10:44:52.261837 14482 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:44:52.259718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60752 (local address 127.23.100.60:36045)
0925 10:44:52.259913 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:52.259917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:52.260552 (+ 635us) server_negotiation.cc:407] Connection header received
0925 10:44:52.260740 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:52.260745 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:52.260818 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:52.260925 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:52.261694 (+ 769us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60752: BlockingRecv error: recv got EOF from 127.0.0.1:60752 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 10:44:52.274236 14408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:52.268927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55024 (local address 127.23.100.61:41659)
0925 10:44:52.269180 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:52.269188 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:52.269675 (+ 487us) server_negotiation.cc:407] Connection header received
0925 10:44:52.269862 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:52.269890 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:52.269947 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:52.270035 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:44:52.270512 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:52.271250 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:52.272535 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:52.272997 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:52.273130 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:52.273549 (+ 419us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:44:52.273640 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:44:52.273744 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:52.273891 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 10:44:52.273962 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
W20260925 10:44:52.285606 14482 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:44:52.283535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60754 (local address 127.23.100.60:36045)
0925 10:44:52.283822 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:52.283831 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:52.284326 (+ 495us) server_negotiation.cc:407] Connection header received
0925 10:44:52.284506 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:52.284512 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:52.284587 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:52.284689 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:52.285394 (+ 705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60754: BlockingRecv error: recv got EOF from 127.0.0.1:60754 (error 108)
Metrics: {"server-negotiator.queue_time_us":147}
I20260925 10:44:52.301789 14408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:52.296155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55036 (local address 127.23.100.61:41659)
0925 10:44:52.296390 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:52.296397 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:52.296901 (+ 504us) server_negotiation.cc:407] Connection header received
0925 10:44:52.297098 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:52.297103 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:52.297198 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:52.297298 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:44:52.297737 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:52.298864 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:52.300114 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:52.300546 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:52.300674 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:52.301057 (+ 383us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:44:52.301161 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:44:52.301255 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:52.301421 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 10:44:52.301486 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260925 10:44:52.309454 14482 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:44:52.307148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60756 (local address 127.23.100.60:36045)
0925 10:44:52.307316 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:52.307320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:52.307957 (+ 637us) server_negotiation.cc:407] Connection header received
0925 10:44:52.308162 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:52.308167 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:52.308271 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:52.308339 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:52.309227 (+ 888us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60756: BlockingRecv error: recv got EOF from 127.0.0.1:60756 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 10:44:52.344373 14408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:52.338735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55040 (local address 127.23.100.61:41659)
0925 10:44:52.339018 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:52.339027 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:52.339572 (+ 545us) server_negotiation.cc:407] Connection header received
0925 10:44:52.339789 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:52.339795 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:52.339875 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:52.339987 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:44:52.340402 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:52.341186 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:52.342599 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:52.343029 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:52.343172 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:52.343619 (+ 447us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:44:52.343742 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:44:52.343850 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:52.344004 (+ 154us) server_negotiation.cc:299] Negotiation successful
0925 10:44:52.344080 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260925 10:44:52.358471 14482 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:44:52.356132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60758 (local address 127.23.100.60:36045)
0925 10:44:52.356394 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:52.356403 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:52.357024 (+ 621us) server_negotiation.cc:407] Connection header received
0925 10:44:52.357222 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:52.357228 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:52.357354 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:52.357467 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:52.358187 (+ 720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60758: BlockingRecv error: recv got EOF from 127.0.0.1:60758 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
I20260925 10:44:52.399281 14408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:52.393480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55052 (local address 127.23.100.61:41659)
0925 10:44:52.393748 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:52.393757 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:52.394281 (+ 524us) server_negotiation.cc:407] Connection header received
0925 10:44:52.394528 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:52.394532 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:52.394615 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:52.394757 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:44:52.395267 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:52.396066 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:52.397436 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:52.397918 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:52.398046 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:52.398465 (+ 419us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:44:52.398582 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:44:52.398681 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:52.398836 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 10:44:52.398912 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260925 10:44:52.410378 14482 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:44:52.408284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60762 (local address 127.23.100.60:36045)
0925 10:44:52.408456 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:52.408464 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:52.408980 (+ 516us) server_negotiation.cc:407] Connection header received
0925 10:44:52.409168 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:52.409173 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:52.409256 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:52.409382 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:52.410129 (+ 747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60762: BlockingRecv error: recv got EOF from 127.0.0.1:60762 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 10:44:52.483253 14408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:52.477512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55060 (local address 127.23.100.61:41659)
0925 10:44:52.477725 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:52.477734 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:52.478277 (+ 543us) server_negotiation.cc:407] Connection header received
0925 10:44:52.478519 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:52.478524 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:52.478604 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:52.478713 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:44:52.479164 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:52.479926 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:52.481174 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:52.481788 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:52.481921 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:52.482355 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:44:52.482522 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:44:52.482630 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:52.482861 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 10:44:52.482937 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260925 10:44:52.490960 14482 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:44:52.488988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60768 (local address 127.23.100.60:36045)
0925 10:44:52.489166 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:52.489172 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:52.489511 (+ 339us) server_negotiation.cc:407] Connection header received
0925 10:44:52.489767 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:52.489772 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:52.489850 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:52.489952 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:52.490726 (+ 774us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60768: BlockingRecv error: recv got EOF from 127.0.0.1:60768 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 10:44:52.633894 14408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:52.628322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55068 (local address 127.23.100.61:41659)
0925 10:44:52.628518 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:52.628524 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:52.628920 (+ 396us) server_negotiation.cc:407] Connection header received
0925 10:44:52.629081 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:52.629086 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:52.629168 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:52.629276 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:44:52.629680 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:52.630676 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:52.631991 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:52.632695 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:52.632814 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:52.633166 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:44:52.633305 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:44:52.633391 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:52.633521 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 10:44:52.633596 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 10:44:52.637061 14485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68f57a46decf45fda1b4213afe411d80 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "68f57a46decf45fda1b4213afe411d80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f57a46decf45fda1b4213afe411d80" member_type: VOTER last_known_addr { host: "127.23.100.62" port: 36181 } } peers { permanent_uuid: "1a0a601dfc024f9bbc826c816b9fb75f" member_type: VOTER last_known_addr { host: "127.23.100.61" port: 41659 } } peers { permanent_uuid: "4ac1f9d63d7042a7a9936a6765af8fc0" member_type: VOTER last_known_addr { host: "127.23.100.60" port: 36045 } } }
I20260925 10:44:52.637398 14485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68f57a46decf45fda1b4213afe411d80 [sys.catalog]: This master's current role is: LEADER
W20260925 10:44:52.642264 14482 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:44:52.640648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60774 (local address 127.23.100.60:36045)
0925 10:44:52.640818 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:52.640822 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:52.641167 (+ 345us) server_negotiation.cc:407] Connection header received
0925 10:44:52.641322 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:52.641325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:52.641416 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:52.641497 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:52.642104 (+ 607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60774: BlockingRecv error: recv got EOF from 127.0.0.1:60774 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 10:44:52.908535 14408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:52.903942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55072 (local address 127.23.100.61:41659)
0925 10:44:52.904128 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:52.904132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:52.904406 (+ 274us) server_negotiation.cc:407] Connection header received
0925 10:44:52.904554 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:52.904558 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:52.904653 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:52.904739 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:44:52.905067 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:52.905738 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:52.907005 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:52.907378 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:52.907467 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:52.907787 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:44:52.907875 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:44:52.907937 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:52.908234 (+ 297us) server_negotiation.cc:299] Negotiation successful
0925 10:44:52.908290 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260925 10:44:52.915683 14482 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:44:52.913999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60784 (local address 127.23.100.60:36045)
0925 10:44:52.914219 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:52.914225 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:52.914430 (+ 205us) server_negotiation.cc:407] Connection header received
0925 10:44:52.914562 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:52.914565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:52.914632 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:52.914708 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:44:52.915455 (+ 747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60784: BlockingRecv error: recv got EOF from 127.0.0.1:60784 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
I20260925 10:44:53.006028 14521 catalog_manager.cc:1259] Loaded cluster ID: 7595e744723845a29469ce351e67113b
I20260925 10:44:53.006189 14521 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4ac1f9d63d7042a7a9936a6765af8fc0: loading cluster ID for follower catalog manager: success
I20260925 10:44:53.008893 14521 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4ac1f9d63d7042a7a9936a6765af8fc0: acquiring CA information for follower catalog manager: success
I20260925 10:44:53.010799 14521 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4ac1f9d63d7042a7a9936a6765af8fc0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 10:44:53.440661 14663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:53.435923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55086 (local address 127.23.100.61:41659)
0925 10:44:53.436383 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:53.436388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:53.436406 (+ 18us) server_negotiation.cc:407] Connection header received
0925 10:44:53.436600 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:53.436604 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:53.436681 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:53.436775 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:44:53.437148 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:53.437898 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:53.439300 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:53.439667 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:53.439762 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:53.440080 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:44:53.440173 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:44:53.440235 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:53.440361 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 10:44:53.440418 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":257,"threads_started":1}
I20260925 10:44:53.451535 14669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:53.445946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60800 (local address 127.23.100.60:36045)
0925 10:44:53.446489 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:53.446496 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:53.446514 (+ 18us) server_negotiation.cc:407] Connection header received
0925 10:44:53.446667 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:53.446672 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:53.446843 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:53.446969 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:44:53.447344 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:53.448024 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:53.449360 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:53.449798 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:53.449969 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:53.450472 (+ 503us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:44:53.450964 (+ 492us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:44:53.451118 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:53.451174 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 10:44:53.451252 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":221,"threads_started":1}
I20260925 10:44:53.467638 14663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:53.457257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55088 (local address 127.23.100.61:41659)
0925 10:44:53.457737 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:53.457743 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:53.461491 (+ 3748us) server_negotiation.cc:407] Connection header received
0925 10:44:53.461731 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:53.461738 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:53.461838 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:53.461899 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:44:53.462682 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:53.463696 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:53.465589 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:53.466183 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:53.466333 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:53.466776 (+ 443us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:44:53.466888 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:44:53.466979 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:53.467229 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 10:44:53.467303 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349}
I20260925 10:44:53.468228 14669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:53.457955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60804 (local address 127.23.100.60:36045)
0925 10:44:53.458141 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:53.458148 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:53.462585 (+ 4437us) server_negotiation.cc:407] Connection header received
0925 10:44:53.462820 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:53.462824 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:53.462890 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:53.462982 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:44:53.463466 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:53.464390 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:53.466218 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:53.466819 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:53.466956 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:53.467532 (+ 576us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:44:53.467681 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:44:53.467761 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:53.467945 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 10:44:53.468020 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 10:44:53.469389 14676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:44:53.457466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46098 (local address 127.23.100.62:36181)
0925 10:44:53.458574 (+ 1108us) server_negotiation.cc:206] Beginning negotiation
0925 10:44:53.458582 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:44:53.462376 (+ 3794us) server_negotiation.cc:407] Connection header received
0925 10:44:53.462667 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:44:53.462672 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:44:53.462893 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:44:53.463056 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:44:53.463808 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:53.464719 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:53.466638 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:44:53.467295 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:44:53.467440 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:44:53.468199 (+ 759us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:44:53.468824 (+ 625us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:44:53.468935 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 10:44:53.469037 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 10:44:53.469115 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":963,"thread_start_us":843,"threads_started":1}
I20260925 10:44:53.474294 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 14263
I20260925 10:44:53.475207 14325 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:44:53.580755 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 14263
W20260925 10:44:53.600957 14430 connection.cc:535] server connection from 127.0.0.1:60668 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 10:44:53.601401 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 14337
I20260925 10:44:53.602530 14399 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:44:53.705324 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 14337
I20260925 10:44:53.724956 23952 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu pid 14411
I20260925 10:44:53.726063 14473 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:44:53.829037 23952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVr7Mel/build/asan/bin/kudu with pid 14411
2026-09-25T10:44:53Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4741 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9463 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (353732 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST